|
| 1 | +# Quick Reference for Vector Development |
| 2 | + |
| 3 | +This guide provides quick commands and coding conventions for Vector development. It's designed to help both AI assistants and human |
| 4 | +contributors get started quickly. |
| 5 | + |
| 6 | +**For comprehensive information, see [CONTRIBUTING.md](CONTRIBUTING.md) and [docs/DEVELOPING.md](docs/DEVELOPING.md).** |
| 7 | + |
| 8 | +## Project Summary |
| 9 | + |
| 10 | +Vector is a high-performance, end-to-end observability data pipeline written in Rust. It collects, transforms, and routes logs, metrics, and |
| 11 | +traces from various sources to any destination. Vector is designed to be reliable, fast, and vendor-neutral, enabling dramatic cost |
| 12 | +reduction and improved data quality for observability infrastructure. |
| 13 | + |
| 14 | +## Project Structure |
| 15 | + |
| 16 | +### Core Directories |
| 17 | + |
| 18 | +- `/src/` - Main Rust source code |
| 19 | + - `sources/` - Data ingestion components |
| 20 | + - `transforms/` - Data processing and routing components |
| 21 | + - `sinks/` - Data output destinations |
| 22 | + - `config/` - Configuration system and validation |
| 23 | + - `topology/` - Component graph management |
| 24 | + - `api/` - GraphQL API for management and monitoring |
| 25 | + - `cli.rs` - Command-line interface |
| 26 | + |
| 27 | +- `/lib/` - Modular library crates |
| 28 | + - `vector-lib/` - Unified library re-exporting core Vector components |
| 29 | + - `vector-core/` - Core event system and abstractions |
| 30 | + - `vector-config/` - Configuration framework with schema generation |
| 31 | + - `vector-buffers/` - Buffering and backpressure management |
| 32 | + - `codecs/` - Data encoding/decoding (JSON, Avro, Protobuf) |
| 33 | + - `enrichment/` - Data enrichment (GeoIP, custom tables) |
| 34 | + - `file-source/` - File watching and reading |
| 35 | + - `prometheus-parser/` - Prometheus metrics parsing |
| 36 | + |
| 37 | +- `/config/` - Configuration examples and templates |
| 38 | +- `/distribution/` - Packaging and deployment configs |
| 39 | + - `docker/` - Docker images (Alpine, Debian, Distroless) |
| 40 | + - `kubernetes/` - Kubernetes manifests |
| 41 | + - `systemd/` - SystemD service files |
| 42 | + - `debian/`, `rpm/` - Linux package configurations |
| 43 | + |
| 44 | +- `/scripts/` - Build, test, and deployment automation |
| 45 | +- `/docs/` - Developer documentation |
| 46 | +- `/tests/` - Integration and E2E tests |
| 47 | + |
| 48 | +## Two Different Workflows |
| 49 | + |
| 50 | +### Rust Development (Most Common) |
| 51 | + |
| 52 | +If you're working on Vector's Rust codebase (sources, sinks, transforms, core functionality): |
| 53 | + |
| 54 | +**Format your code:** |
| 55 | + |
| 56 | +```bash |
| 57 | +make fmt |
| 58 | +``` |
| 59 | + |
| 60 | +**Check formatting:** |
| 61 | + |
| 62 | +```bash |
| 63 | +make check-fmt |
| 64 | +``` |
| 65 | + |
| 66 | +**Run Clippy (linter):** |
| 67 | + |
| 68 | +```bash |
| 69 | +make check-clippy |
| 70 | +``` |
| 71 | + |
| 72 | +**Auto-fix Clippy issues:** |
| 73 | + |
| 74 | +```bash |
| 75 | +make clippy-fix |
| 76 | +``` |
| 77 | + |
| 78 | +**Run unit tests:** |
| 79 | + |
| 80 | +```bash |
| 81 | +make test |
| 82 | +# or |
| 83 | +cargo nextest run --workspace --no-default-features --features "${FEATURES}" |
| 84 | +``` |
| 85 | + |
| 86 | +**Run integration tests:** |
| 87 | + |
| 88 | +```bash |
| 89 | +# See available integration tests: |
| 90 | +# cargo vdev int show |
| 91 | +./scripts/run-integration-test.sh <integration-name> |
| 92 | +``` |
| 93 | + |
| 94 | +See [Integration Tests](#integration-tests) section below for more details. |
| 95 | + |
| 96 | +**Before committing (recommended checks):** |
| 97 | + |
| 98 | +```bash |
| 99 | +make fmt # Format code |
| 100 | +make check-fmt # Verify formatting |
| 101 | +make check-clippy # Run Clippy linter |
| 102 | +make check-component-docs # Check component documentation |
| 103 | +./scripts/check_changelog_fragments.sh # Verify changelog |
| 104 | +``` |
| 105 | + |
| 106 | +### Website/Docs Development (Separate Process) |
| 107 | + |
| 108 | +If you're working on vector.dev website or documentation content: |
| 109 | + |
| 110 | +**Prerequisites:** |
| 111 | + |
| 112 | +- Hugo static site generator |
| 113 | +- CUE CLI tool |
| 114 | +- Node.js and Yarn |
| 115 | +- htmltest |
| 116 | + |
| 117 | +**Run the site locally:** |
| 118 | + |
| 119 | +```bash |
| 120 | +cd website |
| 121 | +make serve |
| 122 | +# Navigate to http://localhost:1313 |
| 123 | +``` |
| 124 | + |
| 125 | +**Build website:** |
| 126 | + |
| 127 | +```bash |
| 128 | +cd website |
| 129 | +make cue-build |
| 130 | +``` |
| 131 | + |
| 132 | +**Note:** Website changes use Hugo, CUE, Tailwind CSS, and TypeScript. See [website/README.md](website/README.md) for details. |
| 133 | + |
| 134 | +## Rust Coding Conventions |
| 135 | + |
| 136 | +### Import Statements (`use`) |
| 137 | + |
| 138 | +All `use` statements must be at the **top of the file/module** or at the top of `mod tests`. |
| 139 | +This is for consistency. |
| 140 | + |
| 141 | +**Correct:** |
| 142 | + |
| 143 | +```rust |
| 144 | +use std::time::Duration; |
| 145 | +use governor::clock; |
| 146 | +use crate::config::TransformConfig; |
| 147 | + |
| 148 | +fn my_function() { |
| 149 | + // function code |
| 150 | +} |
| 151 | +``` |
| 152 | + |
| 153 | +**Incorrect:** |
| 154 | + |
| 155 | +```rust |
| 156 | +fn my_function() { |
| 157 | + use std::time::Duration; // WRONG; Do not insert `use` inside functions |
| 158 | + // function code |
| 159 | +} |
| 160 | +``` |
| 161 | + |
| 162 | +**Organization:** |
| 163 | + |
| 164 | +- Group imports: `std` → external crates → internal (`crate::`) |
| 165 | +- Use `rustfmt` to automatically organize them: `make fmt` |
| 166 | + |
| 167 | +### Logging Style |
| 168 | + |
| 169 | +Always use the [Tracing crate](https://tracing.rs/tracing/)'s key/value style: |
| 170 | + |
| 171 | +**Correct:** |
| 172 | + |
| 173 | +```rust |
| 174 | +warn!(message = "Failed to merge value.", %error); |
| 175 | +info!(message = "Processing batch.", batch_size, internal_log_rate_secs = 1); |
| 176 | +``` |
| 177 | + |
| 178 | +**Incorrect:** |
| 179 | + |
| 180 | +```rust |
| 181 | +warn!("Failed to merge value: {}.", err); // Don't do this |
| 182 | +``` |
| 183 | + |
| 184 | +**Rules:** |
| 185 | + |
| 186 | +- Events should be capitalized and end with a period |
| 187 | +- Use `error` (not `e` or `err`) for error values |
| 188 | +- Prefer Display over Debug: `%error` not `?error` |
| 189 | +- Key/value pairs provide structured logging |
| 190 | + |
| 191 | +### String Formatting |
| 192 | + |
| 193 | +Prefer inline variable syntax in format strings (Rust 1.58+). |
| 194 | + |
| 195 | +**Correct:** |
| 196 | + |
| 197 | +```rust |
| 198 | +format!("Error: {err}"); |
| 199 | +println!("Processing {count} items"); |
| 200 | +``` |
| 201 | + |
| 202 | +**Incorrect:** |
| 203 | + |
| 204 | +```rust |
| 205 | +format!("Error: {}", err); // Unnecessary positional argument |
| 206 | +println!("Processing {} items", count); |
| 207 | +``` |
| 208 | + |
| 209 | +**Why:** Inline syntax is more readable and reduces mistakes with argument ordering. |
| 210 | + |
| 211 | +### Panics |
| 212 | + |
| 213 | +Code in Vector should **NOT** panic under normal circumstances. |
| 214 | + |
| 215 | +- Panics are only acceptable when assumptions about internal state are violated (indicating a bug) |
| 216 | +- All potential panics **MUST** be documented in function documentation |
| 217 | +- Prefer `Result<T, E>` and proper error handling |
| 218 | + |
| 219 | +### Feature Flags |
| 220 | + |
| 221 | +New components (sources, sinks, transforms) must be behind feature flags: |
| 222 | + |
| 223 | +```bash |
| 224 | +# Build only specific component for faster iteration |
| 225 | +cargo test --lib --no-default-features --features sinks-console sinks::console |
| 226 | +``` |
| 227 | + |
| 228 | +See `features` section in `Cargo.toml` for examples. |
| 229 | + |
| 230 | +## Common Patterns |
| 231 | + |
| 232 | +### Development Tools |
| 233 | + |
| 234 | +Vector uses `cargo vdev` for most development tasks. This is a custom CLI tool that wraps common operations: |
| 235 | + |
| 236 | +```bash |
| 237 | +cargo vdev check rust # Clippy |
| 238 | +cargo vdev check fmt # Formatting check |
| 239 | +cargo vdev check events # Event instrumentation check |
| 240 | +cargo vdev check licenses # License compliance |
| 241 | +cargo vdev test # Unit tests |
| 242 | +cargo vdev int test <name> # Integration tests |
| 243 | +cargo vdev fmt # Format code |
| 244 | +``` |
| 245 | + |
| 246 | +### Pre-Push Hook (Optional but Recommended) |
| 247 | + |
| 248 | +Create `.git/hooks/pre-push` with: |
| 249 | + |
| 250 | +```bash |
| 251 | +#!/bin/sh |
| 252 | +set -e |
| 253 | + |
| 254 | +echo "Format code" |
| 255 | +make fmt |
| 256 | + |
| 257 | +echo "Running pre-push checks..." |
| 258 | +make check-licenses |
| 259 | +make check-fmt |
| 260 | +make check-clippy |
| 261 | +make check-component-docs |
| 262 | + |
| 263 | +./scripts/check_changelog_fragments.sh |
| 264 | +``` |
| 265 | + |
| 266 | +Then: `chmod +x .git/hooks/pre-push` |
| 267 | + |
| 268 | +### Container Development |
| 269 | + |
| 270 | +Vector supports development in Docker/Podman containers: |
| 271 | + |
| 272 | +```bash |
| 273 | +ENVIRONMENT=true make <target> |
| 274 | +# Example: ENVIRONMENT=true make test |
| 275 | +``` |
| 276 | + |
| 277 | +## Architecture Notes |
| 278 | + |
| 279 | +### Component Development |
| 280 | + |
| 281 | +- **Sources**: Ingest data from external systems |
| 282 | +- **Transforms**: Modify, filter, or enrich event data |
| 283 | +- **Sinks**: Send data to external systems |
| 284 | + |
| 285 | +Component docs are auto-generated from code annotations. Run `make check-component-docs` after changes. |
| 286 | + |
| 287 | +### Integration Tests |
| 288 | + |
| 289 | +Integration tests verify Vector works with real external services. Require Docker or Podman. |
| 290 | + |
| 291 | +**Run integration tests:** |
| 292 | + |
| 293 | +```bash |
| 294 | +# List available tests |
| 295 | +cargo vdev int show |
| 296 | + |
| 297 | +# Run specific test (example: aws) |
| 298 | +cargo vdev int start aws # need to initiate dev environment first |
| 299 | +cargo vdev int test aws |
| 300 | +``` |
| 301 | + |
| 302 | +See [docs/DEVELOPING.md](docs/DEVELOPING.md#integration-tests) for adding new integration tests. |
| 303 | + |
| 304 | +### Key Files |
| 305 | + |
| 306 | +- `Makefile` - Common build/test/check targets |
| 307 | +- `vdev/` - Custom development CLI tool |
| 308 | +- `src/` - Rust source code |
| 309 | +- `website/` - Hugo-based documentation site |
| 310 | +- `tests/` - Integration and behavior tests |
| 311 | + |
| 312 | +## Common Issues |
| 313 | + |
| 314 | +### Formatting Fails |
| 315 | + |
| 316 | +Run `make fmt` before committing. Formatting must be exact. |
| 317 | + |
| 318 | +### Clippy Errors |
| 319 | + |
| 320 | +Run `make clippy-fix` to auto-fix many issues. Manual fixes may be required. |
| 321 | + |
| 322 | +### Component Docs Out of Sync |
| 323 | + |
| 324 | +Component documentation is generated from code. Run: |
| 325 | + |
| 326 | +```bash |
| 327 | +make check-component-docs |
| 328 | +``` |
| 329 | + |
| 330 | +### License Check Fails |
| 331 | + |
| 332 | +After adding/updating dependencies: |
| 333 | + |
| 334 | +```bash |
| 335 | +cargo install dd-rust-license-tool --locked |
| 336 | +make build-licenses |
| 337 | +``` |
| 338 | + |
| 339 | +## Reference Documentation |
| 340 | + |
| 341 | +These documents provide context that AI agents and developers need when working on Vector code. |
| 342 | + |
| 343 | +### Essential for Code Changes |
| 344 | + |
| 345 | +- **[STYLE.md](STYLE.md)** - Code style rules (formatting, const strings, code organization) |
| 346 | +- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** - System architecture (sources, transforms, sinks, topology) |
| 347 | +- **[docs/DEVELOPING.md](docs/DEVELOPING.md)** - Development workflow and testing |
| 348 | + |
| 349 | +### Component Development |
| 350 | + |
| 351 | +- **[docs/specs/component.md](docs/specs/component.md)** - Component specification (naming, configuration, health checks) |
| 352 | +- **[docs/specs/instrumentation.md](docs/specs/instrumentation.md)** - Instrumentation requirements (event/metric naming) |
| 353 | +- **[src/internal_events](src/internal_events)** - Internal event examples for telemetry |
| 354 | + |
| 355 | +### Adding Documentation |
| 356 | + |
| 357 | +- **[docs/DOCUMENTING.md](docs/DOCUMENTING.md)** - How to document code changes |
| 358 | +- **[changelog.d/README.md](changelog.d/README.md)** - Adding changelog entries |
| 359 | + |
| 360 | +### Full Guides |
| 361 | + |
| 362 | +- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Complete contributing guide |
| 363 | +- **[website/README.md](website/README.md)** - Website development only (separate from Rust code) |
0 commit comments