Skip to content

Commit cdaf42e

Browse files
vsilentvsilentdiaevddependabot[bot]Copilot
authored
* Testing (vsilent#9) * upload artifacts * upload artifacts * syntax fix * try another approach * list files * Update README.md * Add editorconfig. Append gitignore fot emacs * editorconfig * Update README.md * Update actix requirement from 0.10 to 0.11 (vsilent#18) Updates the requirements on [actix](https://github.com/actix/actix) to permit the latest version. - [Release notes](https://github.com/actix/actix/releases) - [Commits](actix/actix@actix-v0.11.0-beta.3...v0.11.1) * Update actix-cors requirement from 0.3.0 to 0.5.4 (vsilent#19) Updates the requirements on [actix-cors](https://github.com/actix/actix-extras) to permit the latest version. - [Release notes](https://github.com/actix/actix-extras/releases) - [Commits](actix/actix-extras@cors-v0.3.0...cors-v0.5.4) * Update bcrypt requirement from 0.8.2 to 0.9.0 (vsilent#17) Updates the requirements on [bcrypt](https://github.com/Keats/rust-bcrypt) to permit the latest version. - [Release notes](https://github.com/Keats/rust-bcrypt/releases) - [Commits](Keats/rust-bcrypt@v0.8.2...v0.9.0) * Update env_logger requirement from 0.7.1 to 0.8.3 (vsilent#16) Updates the requirements on [env_logger](https://github.com/env-logger-rs/env_logger) to permit the latest version. - [Release notes](https://github.com/env-logger-rs/env_logger/releases) - [Changelog](https://github.com/env-logger-rs/env_logger/blob/master/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.7.1...v0.8.3) * Update bigdecimal requirement from 0.0.14 to 0.2.0 (vsilent#15) Updates the requirements on [bigdecimal](https://github.com/akubera/bigdecimal-rs) to permit the latest version. - [Release notes](https://github.com/akubera/bigdecimal-rs/releases) - [Commits](akubera/bigdecimal-rs@v0.0.14...v0.2.0) * Update actix-service requirement from 1.0.6 to 2.0.0 (vsilent#23) Updates the requirements on [actix-service](https://github.com/actix/actix-net) to permit the latest version. - [Release notes](https://github.com/actix/actix-net/releases) - [Commits](actix/actix-net@service-v1.0.6...rt-v2.0.0) * Bump codacy/codacy-analysis-cli-action from 2.0.1 to 3.0.1 (vsilent#24) Bumps [codacy/codacy-analysis-cli-action](https://github.com/codacy/codacy-analysis-cli-action) from 2.0.1 to 3.0.1. - [Release notes](https://github.com/codacy/codacy-analysis-cli-action/releases) - [Commits](codacy/codacy-analysis-cli-action@2.0.1...84fbefe) * Bump codacy/codacy-analysis-cli-action from 3.0.1 to 3.0.2 (vsilent#25) Bumps [codacy/codacy-analysis-cli-action](https://github.com/codacy/codacy-analysis-cli-action) from 3.0.1 to 3.0.2. - [Release notes](https://github.com/codacy/codacy-analysis-cli-action/releases) - [Commits](codacy/codacy-analysis-cli-action@3.0.1...3.0.2) * Bump actions/cache from 2.1.4 to 2.1.5 (vsilent#26) Bumps [actions/cache](https://github.com/actions/cache) from 2.1.4 to 2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...v2.1.5) * Bump codacy/codacy-analysis-cli-action from 3.0.2 to 3.0.3 (vsilent#28) Bumps [codacy/codacy-analysis-cli-action](https://github.com/codacy/codacy-analysis-cli-action) from 3.0.2 to 3.0.3. - [Release notes](https://github.com/codacy/codacy-analysis-cli-action/releases) - [Commits](codacy/codacy-analysis-cli-action@3.0.2...3.0.3) * Update actix requirement from 0.11 to 0.12 (vsilent#31) Updates the requirements on [actix](https://github.com/actix/actix) to permit the latest version. - [Release notes](https://github.com/actix/actix/releases) - [Commits](actix/actix@v0.11.0...v0.12.0) --- updated-dependencies: - dependency-name: actix dependency-type: direct:production ... * Bump actions/cache from 2.1.5 to 2.1.6 (vsilent#29) Bumps [actions/cache](https://github.com/actions/cache) from 2.1.5 to 2.1.6. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.5...v2.1.6) * Update bcrypt requirement from 0.9.0 to 0.10.0 (vsilent#32) Updates the requirements on [bcrypt](https://github.com/Keats/rust-bcrypt) to permit the latest version. - [Release notes](https://github.com/Keats/rust-bcrypt/releases) - [Commits](Keats/rust-bcrypt@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: bcrypt dependency-type: direct:production ... * Bump codacy/codacy-analysis-cli-action from 3.0.3 to 4.0.0 (vsilent#35) Bumps [codacy/codacy-analysis-cli-action](https://github.com/codacy/codacy-analysis-cli-action) from 3.0.3 to 4.0.0. - [Release notes](https://github.com/codacy/codacy-analysis-cli-action/releases) - [Commits](codacy/codacy-analysis-cli-action@3.0.3...4.0.0) --- updated-dependencies: - dependency-name: codacy/codacy-analysis-cli-action dependency-type: direct:production update-type: version-update:semver-major ... * Update env_logger requirement from 0.8.3 to 0.9.0 (vsilent#34) Updates the requirements on [env_logger](https://github.com/env-logger-rs/env_logger) to permit the latest version. - [Release notes](https://github.com/env-logger-rs/env_logger/releases) - [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md) - [Commits](rust-cli/env_logger@v0.8.3...v0.9.0) --- updated-dependencies: - dependency-name: env_logger dependency-type: direct:production ... * Update bigdecimal requirement from 0.2.0 to 0.3.0 (vsilent#37) Updates the requirements on [bigdecimal](https://github.com/akubera/bigdecimal-rs) to permit the latest version. - [Release notes](https://github.com/akubera/bigdecimal-rs/releases) - [Commits](akubera/bigdecimal-rs@v0.2.0...v0.3.0) --- updated-dependencies: - dependency-name: bigdecimal dependency-type: direct:production ... * Update actix-tls requirement from 2.0.0 to 3.0.0 (vsilent#39) Updates the requirements on [actix-tls](https://github.com/actix/actix-net) to permit the latest version. - [Release notes](https://github.com/actix/actix-net/releases) - [Commits](actix/actix-net@rt-v2.0.0...tls-v3.0.0) --- updated-dependencies: - dependency-name: actix-tls dependency-type: direct:production ... * Remove unused imports, list docker containers added * actix-web upgrade * shell commands * shell commands * rustscan, openssl binaries added * rustscan, openssl binaries added * phase 1 files * Broken, integrating bollard for container security check * Update README with new logo and project details Added a new logo image and updated the project description. * Revise README with new images and title case Updated image and title formatting in README. * diesel replaced with r2d2 and rusqlite * ebpf files * refactoring, ebpf / containers * feat(cli): add clap subcommands (serve/sniff) + sniff config - Add clap 4 for CLI argument parsing - Refactor main.rs: dispatch to serve (default) or sniff subcommand - Create src/cli.rs with Cli/Command enums - Create src/sniff/config.rs with SniffConfig (env + CLI args) - Add new deps: clap, async-trait, reqwest, zstd - Update .env.sample with sniff + AI provider config vars - 12 unit tests (7 CLI parsing + 5 config loading) * feat(sniff): log source discovery + database persistence - Create src/sniff/discovery.rs: LogSource, LogSourceType, discovery functions for system logs, Docker containers, and custom paths - Create src/database/repositories/log_sources.rs: CRUD for log_sources and log_summaries tables (follows existing alerts repository pattern) - Add log_sources and log_summaries tables to init_database() - Export docker module from lib.rs for reuse by sniff discovery - 14 unit tests (8 discovery + 6 repository) * feat(sniff): log reader trait + File/Docker/Journald implementations - Create src/sniff/reader.rs with LogReader async trait and LogEntry struct - FileLogReader: byte offset tracking, incremental reads, log rotation detection - DockerLogReader: bollard-based container log streaming with timestamp filtering - JournaldReader: journalctl subprocess (Linux-gated with #[cfg(target_os = "linux")]) - Add futures-util dependency for Docker log stream consumption - 10 unit tests covering read, incremental, truncation, empty lines, metadata * feat(sniff): AI log analysis with OpenAI and pattern backends - Create src/sniff/analyzer.rs with LogAnalyzer trait - OpenAiAnalyzer: single client for OpenAI/Ollama/vLLM/any compatible API sends batched logs to /chat/completions, parses structured JSON response - PatternAnalyzer: fallback local analyzer using regex-free pattern matching detects error spikes, counts errors/warnings without external AI - LogSummary and LogAnomaly types with serialization support - JSON response parsing with graceful handling of partial LLM output - 16 unit tests (prompt building, JSON parsing, pattern analysis, serialization) * feat(sniff): consume mode — zstd compression, dedup, log purge - Create src/sniff/consumer.rs with LogConsumer - FNV hashing deduplication with configurable capacity (100k entries) - zstd compression (level 3) with timestamped archive files - File purge via truncation (preserves fd for syslog daemons) - Docker log purge via /var/lib/docker/containers/ JSON log truncation - Full consume pipeline: deduplicate → compress → purge → report stats - ConsumeResult tracks entries_archived, duplicates_skipped, bytes_freed - 13 unit tests (hashing, dedup, compression, purge, full pipeline) * feat(sniff): reporter + orchestrator loop - Reporter: converts LogSummary/LogAnomaly into Alerts using existing AlertManager infrastructure (route_by_severity, NotificationChannel) - SniffOrchestrator: full discover → read → analyze → report → consume pipeline with continuous and one-shot modes - Wire up run_sniff() in main.rs to use SniffOrchestrator - Add events, rules, alerting, models modules to binary crate - 7 new tests (reporter: 5, orchestrator: 3) * feat(sniff): REST API for log sources and summaries - GET /api/logs/sources — list discovered log sources - POST /api/logs/sources — manually add a custom log source - GET /api/logs/sources/{path} — get a single source - DELETE /api/logs/sources/{path} — remove a source - GET /api/logs/summaries — list AI summaries (optional source_id filter) - Register routes in configure_all_routes - 7 tests covering all endpoints * docs: update CHANGELOG and README for sniff feature - CHANGELOG: document all sniff additions (discovery, readers, AI analysis, consumer, reporter, orchestrator, REST API, deps) - README: add log sniffing to key features, architecture diagram, project structure, CLI usage examples, REST API examples, and completed tasks list * chore: remove task files from repo and gitignore * feat: add curl-based binary installation - install.sh: POSIX shell installer — detects Linux x86_64/aarch64, downloads from GitHub Releases, verifies SHA256, installs to /usr/local/bin - release.yml: GitHub Actions workflow — builds Linux binaries on tag push using cross, creates release with tarballs + checksums - README: add curl install one-liner to Quick Start Usage: curl -fsSL https://raw.githubusercontent.com/vsilent/stackdog/dev/install.sh | sudo bash * docs: fix ML module status — stub infrastructure, not in progress * feat(cli): add --ai-model and --ai-api-url flags to sniff command - Add --ai-model flag to specify AI model (e.g. qwen2.5-coder:latest) - Add --ai-api-url flag to specify API endpoint URL - Recognize "ollama" as AI provider alias (maps to OpenAI-compatible client) - CLI args override env vars for model and API URL - Log AI model and API URL at startup for transparency * feat(sniff): add debug logging and robust LLM JSON extraction - Add debug/trace logging across entire sniff pipeline: discovery, reader, analyzer, orchestrator, reporter - Respect user RUST_LOG env var (no longer hardcoded to info) - Improve LLM response JSON extraction to handle: markdown code fences, preamble text, trailing text - Include raw LLM response in trace logs for debugging parse failures - Show first 200 chars of failed JSON in error messages - Add 5 tests for extract_json edge cases Usage: RUST_LOG=debug stackdog sniff --once ... * feat(alerting): implement real Slack webhook notifications - Add --slack-webhook CLI flag to sniff command - Read STACKDOG_SLACK_WEBHOOK_URL env var (CLI overrides env) - Implement actual HTTP POST to Slack incoming webhook API - Build proper JSON payloads with serde_json (color-coded by severity) - Add reqwest blocking feature for synchronous notification delivery - Wire NotificationConfig through SniffConfig → Orchestrator → Reporter - Add STACKDOG_WEBHOOK_URL env var support - Update .env.sample with notification channel examples - Add 3 tests for Slack webhook config (CLI, env, override priority) Usage: stackdog sniff --once --slack-webhook https://hooks.slack.com/services/T/B/xxx # or via env: export STACKDOG_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T/B/xxx * Update docker.yml --------- Co-authored-by: vsilent <jabberroid@gmail.com> Co-authored-by: Evgeny Duzhakov <diaevd@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent f750dce commit cdaf42e

41 files changed

Lines changed: 5775 additions & 106 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.env.sample

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,21 @@ APP_HOST=0.0.0.0
55
APP_PORT=5000
66
DATABASE_URL=stackdog.db
77
RUST_BACKTRACE=full
8+
9+
# Log Sniff Configuration
10+
#STACKDOG_LOG_SOURCES=/var/log/syslog,/var/log/auth.log
11+
#STACKDOG_SNIFF_INTERVAL=30
12+
#STACKDOG_SNIFF_OUTPUT_DIR=./stackdog-logs/
13+
14+
# AI Provider Configuration
15+
# Supports OpenAI, Ollama (http://localhost:11434/v1), or any OpenAI-compatible API
16+
#STACKDOG_AI_PROVIDER=openai
17+
#STACKDOG_AI_API_URL=http://localhost:11434/v1
18+
#STACKDOG_AI_API_KEY=
19+
#STACKDOG_AI_MODEL=llama3
20+
21+
# Notification Channels
22+
# Slack: create an incoming webhook at https://api.slack.com/messaging/webhooks
23+
#STACKDOG_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T.../B.../xxxxx
24+
# Generic webhook endpoint for alert notifications
25+
#STACKDOG_WEBHOOK_URL=https://example.com/webhook

.github/copilot-instructions.md

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Stackdog Security — Copilot Instructions
2+
3+
## What This Project Is
4+
5+
Stackdog is a Rust-based security platform for Docker containers and Linux servers. It collects events via eBPF syscall monitoring, runs them through a rule/signature engine and optional ML anomaly detection, manages firewall responses (nftables/iptables + container quarantine), and exposes a REST + WebSocket API consumed by a React/TypeScript dashboard.
6+
7+
## Workspace Structure
8+
9+
This is a Cargo workspace with two crates:
10+
- `.` — Main crate (`stackdog`): HTTP server, all security logic
11+
- `ebpf/` — Separate crate (`stackdog-ebpf`): eBPF programs compiled for the kernel (uses `aya-ebpf`)
12+
13+
## Build, Test, and Lint Commands
14+
15+
```bash
16+
# Build
17+
cargo build
18+
cargo build --release
19+
20+
# Tests
21+
cargo test --lib # Unit tests only (in-source)
22+
cargo test --all # All tests including integration
23+
cargo test --lib -- events:: # Run tests for a specific module
24+
cargo test --lib -- rules::scorer # Run a single test by name prefix
25+
26+
# Code quality
27+
cargo fmt --all
28+
cargo clippy --all
29+
cargo audit # Dependency vulnerability scan
30+
31+
# Benchmarks
32+
cargo bench
33+
34+
# Frontend (in web/)
35+
npm test
36+
npm run lint
37+
npm run build
38+
```
39+
40+
## Environment Setup
41+
42+
Requires a `.env` file (copy `.env.sample`). Key variables:
43+
```
44+
APP_HOST=0.0.0.0
45+
APP_PORT=5000
46+
DATABASE_URL=stackdog.db
47+
RUST_BACKTRACE=full
48+
```
49+
50+
System dependencies (Linux): `libsqlite3-dev libssl-dev clang llvm pkg-config`
51+
52+
## Architecture
53+
54+
```
55+
Collectors (Linux only) Rule Engine Response
56+
eBPF syscall events → Signatures → nftables/iptables
57+
Docker daemon events → Threat scoring → Container quarantine
58+
Network events → ML anomaly det. → Alerting
59+
60+
REST + WebSocket API
61+
React/TypeScript UI
62+
```
63+
64+
**Key src/ modules:**
65+
66+
| Module | Purpose |
67+
|---|---|
68+
| `events/` | Core event types: `SyscallEvent`, `SecurityEvent`, `NetworkEvent`, `ContainerEvent` |
69+
| `rules/` | Rule engine, signature database, threat scorer |
70+
| `alerting/` | `AlertManager`, notification channels (Slack/email/webhook) |
71+
| `collectors/` | eBPF loader, Docker daemon events, network collector (Linux only) |
72+
| `firewall/` | nftables management, iptables fallback, `QuarantineManager` (Linux only) |
73+
| `ml/` | Candle-based anomaly detection (optional `ml` feature) |
74+
| `correlator/` | Event correlation engine |
75+
| `baselines/` | Baseline learning for anomaly detection |
76+
| `database/` | SQLite connection pool (`r2d2` + raw `rusqlite`), repositories |
77+
| `api/` | actix-web REST endpoints + WebSocket |
78+
| `response/` | Automated response action pipeline |
79+
80+
## Key Conventions
81+
82+
### Platform-Gating
83+
Linux-only modules (`collectors`, `firewall`) and deps (aya, netlink) are gated:
84+
```rust
85+
#[cfg(target_os = "linux")]
86+
pub mod firewall;
87+
```
88+
The `ebpf` and `ml` features are opt-in and must be enabled explicitly:
89+
```bash
90+
cargo build --features ebpf
91+
cargo build --features ml
92+
```
93+
94+
### Error Handling
95+
- Use `anyhow::{Result, Context}` for application/binary code
96+
- Use `thiserror` for library error types
97+
- Never use `.unwrap()` in production code; use `?` with `.context("...")`
98+
99+
### Database
100+
The project uses raw `rusqlite` with `r2d2` connection pooling. `DbPool` is `r2d2::Pool<SqliteConnectionManager>`. Tables are created with `CREATE TABLE IF NOT EXISTS` in `database::connection::init_database`. Repositories are in `src/database/repositories/` and receive a `&DbPool`.
101+
102+
### API Routes
103+
Each API sub-module exports a `configure_routes(cfg: &mut web::ServiceConfig)` function. All routes are composed in `api::configure_all_routes`, which is the single call site in `main.rs`.
104+
105+
### Test Location
106+
- **Unit tests**: `#[cfg(test)] mod tests { ... }` inside source files
107+
- **Integration tests**: `tests/` directory at workspace root
108+
109+
### eBPF Programs
110+
The `ebpf/` crate is compiled separately for the Linux kernel. User-space loading is handled by `src/collectors/ebpf/` using the `aya` library. Kernel-side programs use `aya-ebpf`.
111+
112+
### Async Runtime
113+
The main binary uses `#[actix_rt::main]`. Library code uses `tokio`. Avoid mixing runtimes.

.github/workflows/docker.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@ on:
1212
jobs:
1313
cicd-linux-docker:
1414
name: Cargo and npm build
15-
runs-on: ubuntu-latest
15+
#runs-on: ubuntu-latest
16+
runs-on: [self-hosted, linux]
1617
steps:
1718
- name: Checkout sources
1819
uses: actions/checkout@v2
@@ -135,7 +136,8 @@ jobs:
135136

136137
cicd-docker:
137138
name: CICD Docker
138-
runs-on: ubuntu-latest
139+
#runs-on: ubuntu-latest
140+
runs-on: [self-hosted, linux]
139141
needs: cicd-linux-docker
140142
steps:
141143
- name: Download app archive

.github/workflows/release.yml

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Release
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*"
7+
8+
permissions:
9+
contents: write
10+
11+
env:
12+
CARGO_TERM_COLOR: always
13+
14+
jobs:
15+
build:
16+
name: Build ${{ matrix.target }}
17+
runs-on: ubuntu-latest
18+
strategy:
19+
matrix:
20+
include:
21+
- target: x86_64-unknown-linux-gnu
22+
artifact: stackdog-linux-x86_64
23+
- target: aarch64-unknown-linux-gnu
24+
artifact: stackdog-linux-aarch64
25+
26+
steps:
27+
- name: Checkout
28+
uses: actions/checkout@v4
29+
30+
- name: Install Rust toolchain
31+
uses: dtolnay/rust-toolchain@stable
32+
with:
33+
targets: ${{ matrix.target }}
34+
35+
- name: Install cross
36+
run: cargo install cross --git https://github.com/cross-rs/cross
37+
38+
- name: Build release binary
39+
run: cross build --release --target ${{ matrix.target }}
40+
41+
- name: Package
42+
run: |
43+
mkdir -p dist
44+
cp target/${{ matrix.target }}/release/stackdog dist/stackdog
45+
cd dist
46+
tar czf ${{ matrix.artifact }}.tar.gz stackdog
47+
sha256sum ${{ matrix.artifact }}.tar.gz > ${{ matrix.artifact }}.tar.gz.sha256
48+
49+
- name: Upload artifact
50+
uses: actions/upload-artifact@v4
51+
with:
52+
name: ${{ matrix.artifact }}
53+
path: |
54+
dist/${{ matrix.artifact }}.tar.gz
55+
dist/${{ matrix.artifact }}.tar.gz.sha256
56+
57+
release:
58+
name: Create GitHub Release
59+
needs: build
60+
runs-on: ubuntu-latest
61+
steps:
62+
- name: Checkout
63+
uses: actions/checkout@v4
64+
65+
- name: Download all artifacts
66+
uses: actions/download-artifact@v4
67+
with:
68+
path: artifacts
69+
merge-multiple: true
70+
71+
- name: Create release
72+
uses: softprops/action-gh-release@v2
73+
with:
74+
generate_release_notes: true
75+
files: |
76+
artifacts/*.tar.gz
77+
artifacts/*.sha256

.gitignore

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,4 @@ Cargo.lock
3333
# End of https://www.gitignore.io/api/rust,code
3434

3535
.idea
36-
<<<<<<< HEAD
37-
=======
38-
*.db
39-
>>>>>>> testing
36+
docs/tasks/

0 commit comments

Comments
 (0)