diff --git a/.gitignore b/.gitignore index d78e95f..7a1a399 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,7 @@ !LICENSE !LICENSE_COMMERCIAL !CONTRIBUTING.md +!CRATE_USAGE.md # Whitelist src directory and all Rust files within it !src/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e9d607a..ba736b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,6 +32,12 @@ By submitting a pull request or otherwise contributing code to this project, you - Write meaningful commit messages. - Document all public API functions. - Add comments for complex or non-obvious code sections. +- Enable the pre-commit formatting hook (recommended): + +```bash +cp .github/pre-commit-hook.sh .git/hooks/pre-commit +chmod +x .git/hooks/pre-commit +``` ## Reporting Issues diff --git a/CRATE_USAGE.md b/CRATE_USAGE.md new file mode 100644 index 0000000..9d350ec --- /dev/null +++ b/CRATE_USAGE.md @@ -0,0 +1,96 @@ +# Crate Usage: bbl_parser + +Focused guidance for using the bbl_parser Rust crate. + +## Table of Contents +- [Installation](#installation) +- [Cargo features](#cargo-features) +- [Basic usage](#basic-usage) +- [Multi-log processing](#multi-log-processing) +- [Parsing from memory](#parsing-from-memory) +- [Examples](#examples) +- [Notes](#notes) + +## Installation + +Add the dependency to your Cargo.toml: + +```toml +[dependencies] +# Local path while developing; use a version when published +bbl_parser = { path = "path/to/bbl_parser" } +``` + +## Cargo features + +- `csv` (default): CSV export helpers +- `cli` (default): Command-line entry points +- `json`: JSON-related helpers (requires `serde`) +- `serde`: Enable serialization for data structures + +If you only need the parser types and functions, the defaults are fine. + +## Basic usage + +```rust +use bbl_parser::{parse_bbl_file, ExportOptions}; +use std::path::Path; + +fn main() -> anyhow::Result<()> { + let log = parse_bbl_file(Path::new("flight.BBL"), ExportOptions::default(), false)?; + println!("firmware: {}", log.header.firmware_revision); + println!("frames: {}", log.sample_frames.len()); + Ok(()) +} +``` + +Key outputs on the BBLLog: +- `header`: configuration and metadata +- `sample_frames`: decoded I/P/S/G/H/E frames +- `event_frames`: flight events (when present) +- `gps_track`: GPS coordinates (when present) + +## Multi-log processing + +```rust +use bbl_parser::{parse_bbl_file_all_logs, ExportOptions}; +use std::path::Path; + +fn main() -> anyhow::Result<()> { + let logs = parse_bbl_file_all_logs(Path::new("multi_flight.BBL"), ExportOptions::default(), false)?; + for log in logs { + println!("log {} of {} -> frames {}", log.log_number, log.total_logs, log.sample_frames.len()); + } + Ok(()) +} +``` + +## Parsing from memory + +```rust +use bbl_parser::{parse_bbl_bytes, ExportOptions}; + +fn main() -> anyhow::Result<()> { + let bytes = std::fs::read("flight.BBL")?; + let log = parse_bbl_bytes(&bytes, ExportOptions::default(), false)?; + println!("frames: {}", log.sample_frames.len()); + Ok(()) +} +``` + +## Examples + +Run the crate example that demonstrates multi-firmware support and PID extraction: + +```bash +cargo build --example bbl_crate_test +cargo run --example bbl_crate_test -- flight.BBL +``` + +More details: [examples/README.md](./examples/README.md) + +## Notes + +- API is evolving while the project is WIP; names and structures may change. +- CSV field order and naming follow blackbox-tools to maximize compatibility. +- For CLI usage and high-level overview, see the main [README](./README.md). diff --git a/OVERVIEW.md b/OVERVIEW.md index 54c41fa..dcc5184 100644 --- a/OVERVIEW.md +++ b/OVERVIEW.md @@ -77,55 +77,63 @@ The BBL parser implements a streaming architecture designed for memory efficienc - **E-frames:** Flight events with official Betaflight FlightLogEvent enum mapping ### **Export Functionality** -- **CSV Export:** blackbox_decode compatible format with proper field ordering -- **GPX Export:** Standard GPS exchange format for mapping applications -- **Event Export:** JSONL format with Betaflight event type descriptions +- **CSV Export:** blackbox_decode compatible format with proper field ordering (CLI functional, crate stub) +- **GPX Export:** Standard GPS exchange format for mapping applications (CLI functional) +- **Event Export:** JSONL format with Betaflight event type descriptions (CLI functional) + +**Note:** Export functionality is currently implemented in the CLI (`src/main.rs`). Crate-level export functions in `src/export.rs` are stubs pending systematic migration. ### **Encoding Support** BBL encoding compatibility: `SIGNED_VB`, `UNSIGNED_VB`, `NEG_14BIT`, `TAG8_8SVB`, `TAG2_3S32`, `TAG8_4S16` ### **Project Structure** -``` +```text src/ -├── main.rs # CLI interface, file handling, statistics +├── main.rs # CLI interface, file handling, statistics, CSV export +├── lib.rs # Library API exports and documentation ├── bbl_format.rs # BBL binary format decoding and encoding +├── conversion.rs # Unit conversions (GPS coordinates, altitude, speed) ├── error.rs # Error handling and result types +├── export.rs # Export function stubs (CSV/GPX/Event migration in progress) ├── types/ # Core data structures -│ ├── mod.rs # Module definitions -│ ├── log.rs # Log container types -│ ├── frame.rs # Frame data structures -│ └── header.rs # Header information types +│ ├── mod.rs # Module definitions and re-exports +│ ├── log.rs # BBLLog container type +│ ├── frame.rs # DecodedFrame and FrameDefinition structures +│ ├── header.rs # BBLHeader and field definitions +│ └── gps.rs # GpsCoordinate, GpsHomeCoordinate, EventFrame └── parser/ # Parsing implementation ├── mod.rs # Parser module definitions - ├── decoder.rs # Frame decoding logic - ├── frame.rs # Frame parsing implementations + ├── main.rs # High-level parsing entry points (parse_bbl_file, parse_bbl_bytes) + ├── decoder.rs # Frame decoding logic and predictors + ├── frame.rs # Frame parsing implementations (I, P, S, G, H, E frames) ├── header.rs # Header parsing logic - └── stream.rs # Binary stream handling + └── stream.rs # Binary stream handling (BBLDataStream) ``` --- ## 🚀 **Current Features** -### **Current Features** - ### **File Processing** - **Universal Format Support:** `.BBL`, `.BFL`, `.TXT` with case-insensitive matching - **Firmware Compatibility:** Betaflight, EmuFlight, INAV support - **Multi-log Processing:** Automatic detection of multiple flight sessions in single files +- **Smart Export Filtering:** Automatically skips short test flights (<5s) while preserving high-quality short logs ### **Library API** - **Complete Data Access:** Programmatic access to all BBL data structures -- **Memory-Based Parsing:** Parse from file paths or memory buffers -- **Multi-Log Support:** Handle files containing multiple flight sessions +- **Memory-Based Parsing:** Parse from file paths or memory buffers (`parse_bbl_file`, `parse_bbl_bytes`) +- **Multi-Log Support:** Handle files containing multiple flight sessions (`parse_bbl_file_all_logs`, `parse_bbl_bytes_all_logs`) - **Serde Integration:** Optional serialization support for data structures - **Rust Crate:** Available as library dependency for 3rd party projects -### **Data Export Capabilities** +### **Data Export Capabilities (CLI)** - **CSV Export:** blackbox_decode compatible field ordering and formatting -- **GPS Export:** GPX format generation for mapping applications -- **Event Export:** Flight event data in JSONL format with Betaflight event type mapping -- **Multi-log Support:** Individual files for each flight session + - Main flight data `[.XX].csv` with proper field order and "time (us)" column + - Headers `[.XX].headers.csv` with complete configuration +- **GPS Export:** GPX format generation for mapping applications (`[.XX].gps.gpx`) +- **Event Export:** Flight event data in JSONL format (`[.XX].event`) +- **Multi-log Support:** Individual numbered files for each flight session (`.01.`, `.02.`, etc.) - **Metadata Export:** Complete header and configuration information ### **Architecture Benefits** @@ -165,35 +173,9 @@ src/ --- -## 🔍 **Current Data Processing** - -### **Frame Processing** -- **Frame Parsing:** Basic parsing of all major frame types -- **Temporal Resolution:** Maintains flight sequence timing -- **Error Detection:** Basic validation with diagnostic output -- **Memory Management:** Streaming architecture for large files - -### **Export Quality** -- **CSV Compatibility:** Basic blackbox_decode format compatibility -- **GPS Accuracy:** Coordinate conversion with firmware-specific scaling -- **Event Mapping:** Official Betaflight FlightLogEvent enum compliance - ---- - -## 📈 **Development Focus Areas** - -### **Current Priorities** -- **Testing:** Comprehensive validation across firmware versions -- **Performance:** Optimization for large file processing -- **Error Handling:** Improved diagnostic and recovery capabilities -- **Documentation:** Complete API documentation and usage examples -- **Compatibility:** Enhanced support for edge cases and unusual file formats - ---- - ## 📋 **Usage Examples** -### **Basic Processing** +### **Command-Line Interface** ```bash # Single file analysis ./target/release/bbl_parser flight.BBL @@ -213,11 +195,14 @@ src/ # All export formats ./target/release/bbl_parser --csv --gpx --event logs/*.BBL +# Force export all logs (bypasses smart filtering) +./target/release/bbl_parser --csv --force-export logs/*.BBL + # Debug mode for development analysis ./target/release/bbl_parser --debug problematic_file.BBL ``` -### **Typical Output** +### **Typical CLI Output** ``` Processing: BTFL_BLACKBOX_LOG_20250601_121852.BBL @@ -243,21 +228,6 @@ Exported event data to: BTFL_BLACKBOX_LOG_20250601_121852.event --- -## 🔍 **Current Data Processing** - -### **Frame Processing** -- **Frame Parsing:** Basic parsing of all major frame types -- **Temporal Resolution:** Maintains flight sequence timing -- **Error Detection:** Basic validation with diagnostic output -- **Memory Management:** Streaming architecture for large files - -### **Export Quality** -- **CSV Compatibility:** Basic blackbox_decode format compatibility -- **GPS Accuracy:** Coordinate conversion with firmware-specific scaling -- **Event Mapping:** Official Betaflight FlightLogEvent enum compliance - ---- - ## 🎯 **Use Cases & Applications** ### **Current Applications** @@ -280,13 +250,18 @@ Exported event data to: BTFL_BLACKBOX_LOG_20250601_121852.event ## 📝 **Documentation** ### **Available Documentation** -- **README.md** - User guide, installation, usage examples, and complete library API documentation -- **OVERVIEW.md** - Technical architecture and feature overview +- **README.md** - CLI-focused user guide with installation and quick start +- **CRATE_USAGE.md** - Rust crate API usage guide with code examples +- **OVERVIEW.md** - Technical architecture and feature overview (this document) - **FRAMES.md** - Frame format specifications and encoding details - **GOALS.md** - Project objectives and design principles +- **examples/README.md** - Example programs demonstrating crate usage ### **Development Documentation** -API documentation available via `cargo doc` and comprehensive usage examples in README.md for library integration. +- API documentation available via `cargo doc` +- Comprehensive crate usage examples in `CRATE_USAGE.md` and `examples/` +- Pre-commit hooks for automatic code formatting (`.github/pre-commit-hook.sh`) +- Development environment setup script (`.github/setup-dev.sh`) --- diff --git a/README.md b/README.md index 3059755..3bcbeff 100644 --- a/README.md +++ b/README.md @@ -1,532 +1,86 @@ -# BBL Parser v0.9.0 (Work-in-Progress) +# BBL Parser (Work-in-Progress) -A high-performance Rust library and command-line tool for parsing BBL (Blackbox Log) files from flight controllers. +A fast, pure-Rust Blackbox Log parser primarily used as a command-line tool, with an optional Rust crate API. -**Version:** 0.9.0 🚧 **Work-in-Progress** -**Status:** Under active development -**Supported Formats:** `.BBL`, `.BFL`, `.TXT` (case-insensitive) - Compatible with Betaflight, EmuFlight, and INAV -**Library API:** Provides complete in-memory access to flight data, headers, GPS coordinates, and events +Supports `.BBL`, `.BFL`, `.TXT` (case-insensitive) across Betaflight, EmuFlight, and INAV. -## Current Features +## Table of Contents +- [Overview](#overview) +- [Features](#features) +- [Quick start (cli)](#quick-start-cli) +- [Output formats](#output-formats) +- [Smart export filtering](#smart-export-filtering) +- [Documentation](#documentation) +- [License](#license) +- [Acknowledgments](#acknowledgments) -- **Rust Library API**: Complete programmatic access to BBL data structures in memory -- **Pure Rust Implementation**: Direct parsing logic without external dependencies -- **Universal File Support**: All common BBL formats with case-insensitive extension matching -- **Complete Frame Support**: I, P, H, S, E, G frames with all encoding formats (SIGNED_VB, UNSIGNED_VB, NEG_14BIT, TAG8_8SVB, TAG2_3S32, TAG8_4S16) -- **Multi-Log Processing**: Automatic detection and processing of multiple flight logs within single files -- **Streaming Architecture**: Memory-efficient processing for large files (500K+ frames) -- **Advanced Frame Prediction**: Full predictor implementation for accurate P-frame decoding -- **CSV Export**: Flight data and header export with blackbox_decode compatibility -- **GPS Export**: GPX file generation for GPS-enabled flight logs -- **Event Export**: Flight event data extraction in JSONL format -- **Smart Export Filtering**: Automatically skips short test flights while preserving high-quality short logs -- **Command Line Interface**: Glob patterns, debug mode, configurable output directories -- **Comprehensive Examples**: Practical demonstrations of crate usage with PID display and multi-firmware support - -## Export Formats - -### CSV Export (`--csv`) - -Exports blackbox logs to CSV format with blackbox_decode compatibility: - -- **`[.XX].csv`**: Main flight data file containing I, P, S, G frame data - - Field names header row in blackbox_decode compatible order - - Time field labeled as "time (us)" for microsecond precision - - All flight loop data (I frames) and status data (S frames) - - GPS data (G frames) when available - - Time-sorted chronological data rows -- **`[.XX].headers.csv`**: Complete header information file - - Field,Value format with all configuration parameters - - Frame definitions, system settings, firmware information - - All BBL header metadata for analysis tools - -### GPS Export (`--gpx`) - -Exports GPS data to GPX format for mapping applications: - -- **`[.XX].gps.gpx`**: GPS track file in standard GPX format - - Geographic coordinates from GPS frames - - Altitude information with proper firmware scaling - - Timestamp data for track visualization - - Compatible with Google Earth, GPS visualization tools - -### Event Export (`--event`) - -Exports flight events to JSONL format: - -- **`[.XX].event`**: Flight event data in JSON Lines format - - Individual JSON objects per line for streaming compatibility - - Event types based on official Betaflight FlightLogEvent enum - - Includes sync beeps, disarm events, flight mode changes, log boundaries - - Compatible with log analysis tools expecting JSONL format - -**Filename Logic:** -- **Single log files**: Clean filenames without numbering (`flight_log.csv`, `flight_log.event`, `flight_log.gps.gpx`) -- **Multiple log files**: Numbered sequence for clarity (`flight_log.01.csv`, `flight_log.02.event`, etc.) - -**Example files generated:** - -*Single log in BBL file:* -``` -BTFL_LOG_20250601_121852.csv # Flight data -BTFL_LOG_20250601_121852.headers.csv # Headers -BTFL_LOG_20250601_121852.gps.gpx # GPS track data -BTFL_LOG_20250601_121852.event # Flight events -``` - -*Multiple logs in BBL file:* -``` -BTFL_LOG_20250601_121852.01.csv # Flight data for log 1 -BTFL_LOG_20250601_121852.01.headers.csv # Headers for log 1 -BTFL_LOG_20250601_121852.01.gps.gpx # GPS data for log 1 -BTFL_LOG_20250601_121852.01.event # Events for log 1 -BTFL_LOG_20250601_121852.02.csv # Flight data for log 2 -BTFL_LOG_20250601_121852.02.headers.csv # Headers for log 2 -BTFL_LOG_20250601_121852.02.gps.gpx # GPS data for log 2 -BTFL_LOG_20250601_121852.02.event # Events for log 2 -``` - -## Installation & Usage - -### As a Library (Rust Crate) - -Add `bbl_parser` to your `Cargo.toml`: - -```toml -[dependencies] -bbl_parser = { path = "path/to/bbl_parser" } -# or when published to crates.io: -# bbl_parser = "0.9.0" - -# Optional features: -bbl_parser = { version = "0.9.0", features = ["serde", "json"] } -``` - -#### Crate Features - -- **`csv`** (default): CSV export functionality -- **`json`**: JSON export support (requires `serde`) -- **`serde`**: Serialization support for data structures -- **`cli`** (default): Command line interface support - -#### Basic Library Usage - -```rust -use bbl_parser::{parse_bbl_file, ExportOptions, BBLLog}; -use std::path::Path; - -fn main() -> anyhow::Result<()> { - // Parse BBL file into memory structures - let export_options = ExportOptions::default(); // No file exports - let log = parse_bbl_file( - Path::new("flight.BBL"), - export_options, - false // debug mode - )?; - - // Access header information - println!("Firmware: {}", log.header.firmware_revision); - println!("Board: {}", log.header.board_info); - println!("Craft: {}", log.header.craft_name); - - // Access frame data - println!("Total frames: {}", log.sample_frames.len()); - for frame in &log.sample_frames { - println!("Frame {}: {} fields at {}μs", - frame.frame_type, - frame.data.len(), - frame.timestamp_us); - } - - // Access GPS data - for gps in &log.gps_coordinates { - println!("GPS: lat={}, lon={}, alt={}m", - gps.latitude, gps.longitude, gps.altitude); - } - - // Access flight events - for event in &log.event_frames { - println!("Event type: {} at {}μs", - event.event_type, event.timestamp_us); - } - - Ok(()) -} -``` - -#### Available Data Structures - -The library exposes these main structures for 3rd party access: - -> **Note:** This crate is currently undergoing systematic migration to a proper library API. Some data structures and field names may change in future versions as the migration completes. - -**`BBLLog`** - Main container for all log data: -```rust -pub struct BBLLog { - pub log_number: usize, // Log number (1, 2, 3...) - pub total_logs: usize, // Total logs in file - pub header: BBLHeader, // Header/configuration data - pub stats: FrameStats, // Frame counts and timing - pub sample_frames: Vec, // Main flight data - pub gps_coordinates: Vec, // GPS track points - pub home_coordinates: Vec, // Home position - pub event_frames: Vec, // Flight events -} -``` - -**`BBLHeader`** - Configuration and metadata: -```rust -pub struct BBLHeader { - pub firmware_revision: String, // "Betaflight 4.5.2 (024f8e13d)" - pub board_info: String, // "AXFL AXISFLYINGF7PRO" - pub craft_name: String, // "My Quad" - pub data_version: u8, // BBL format version - pub looptime: u32, // Main loop time in μs - pub sysconfig: HashMap, // All system parameters -} -``` - -**`DecodedFrame`** - Individual flight data points: -```rust -pub struct DecodedFrame { - pub frame_type: char, // 'I', 'P', 'S', 'G', 'H', 'E' - pub timestamp_us: u64, // Time in microseconds - pub data: HashMap, // Field name -> value mapping -} -``` - -**`GpsCoordinate`** - GPS position data: -```rust -pub struct GpsCoordinate { - pub latitude: f64, // Decimal degrees - pub longitude: f64, // Decimal degrees - pub altitude: i32, // Altitude in meters - pub timestamp_us: u64, // Time in microseconds -} -``` - -**`EventFrame`** - Flight events and state changes: -```rust -pub struct EventFrame { - pub timestamp_us: u64, // Time in microseconds - pub event_type: u8, // Event type ID - pub event_name: String, // Human-readable name - pub data: Option, // Optional event data -} -``` - -#### Multi-Log Processing - -```rust -use bbl_parser::parse_bbl_file_all_logs; - -// Parse file with multiple flight logs -let logs = parse_bbl_file_all_logs( - Path::new("multi_flight.BBL"), - ExportOptions::default(), - false -)?; - -for log in logs { - println!("Processing log {} of {}", log.log_number, log.total_logs); - println!("Duration: {:.1}s", log.duration_seconds()); - println!("Frames: {}", log.sample_frames.len()); -} -``` - -#### Memory-Based Parsing - -```rust -use bbl_parser::parse_bbl_bytes; - -// Parse BBL data from memory (Vec, &[u8]) -let bbl_data: Vec = std::fs::read("flight.BBL")?; -let log = parse_bbl_bytes(&bbl_data, ExportOptions::default(), false)?; -``` - -#### Examples - -**BBL Crate Test Example** (`examples/bbl_crate_test`) +## Overview -A comprehensive demonstration of the BBL parser crate featuring: -- Multi-firmware support (Betaflight, EmuFlight, iNav) -- PID settings extraction with feedforward values -- Multi-log file processing with glob patterns -- Flight duration calculation and statistics +BBL Parser reads flight controller blackbox logs and provides a command-line interface to export CSV, GPX, and event data. -```bash -# Build the example -cargo build --example bbl_crate_test +A Rust crate API is also available for programmatic access -- see [CRATE_USAGE.md](./CRATE_USAGE.md). -# Run with a single file -cargo run --example bbl_crate_test -- flight.BBL +The CSV export matches blackbox-tools field order and naming, and decoding includes full P-frame predictor logic. -# Run with multiple files or patterns -cargo run --example bbl_crate_test -- logs/*.BBL *.bbl -``` +## Features -See [`examples/README.md`](examples/README.md) for detailed usage instructions. +- Pure Rust parser (no external binaries) +- Command-line interface (CLI) +- Multi-log file support +- I, P, H, S, G, E frame decoding (reference-compliant) +- CSV export compatible with blackbox_decode +- GPX export for GPS tracks +- Event export (CLI) +- Streaming architecture suitable for large logs -### As a Command Line Tool +## Quick start (cli) ```bash -git clone -cd bbl_parser +# Build once cargo build --release -``` -**Note:** The main CLI tool provides CSV/GPX export and analysis. For a simpler demonstration of crate usage with PID display, see the `examples/bbl_crate_test` example above. +# Analyze a file (console stats only) +./target/release/bbl_parser flight.BBL -### Basic Usage -```bash -# Analysis and console statistics only -./target/release/bbl_parser file.BBL - -# CSV export -./target/release/bbl_parser --csv logs/*.BBL - -# GPS export to GPX format (console stats + GPX file) -./target/release/bbl_parser --gpx logs/*.BBL - -# Event data export (console stats + event file) -./target/release/bbl_parser --event logs/*.BBL - -# All export formats +# Export CSV / GPX / Events ./target/release/bbl_parser --csv --gpx --event logs/*.BBL -# Multiple files and formats -./target/release/bbl_parser file1.BBL file2.BFL file3.TXT - -# Glob patterns -./target/release/bbl_parser logs/*.{BBL,BFL,TXT} - -# Debug mode -./target/release/bbl_parser --debug logs/*.BBL - -# Custom output directory -./target/release/bbl_parser --csv --output-dir ./output logs/*.BBL - -# Force export all logs (bypasses smart filtering) -./target/release/bbl_parser --csv --force-export logs/*.BBL -``` - -### Smart Export Filtering - -By default, the tool uses smart filtering to skip short test flights and focus on meaningful flight data: - -- **< 5 seconds**: Always skipped (brief arming, connectivity tests) -- **5-15 seconds**: Kept only if data density > 1500 fps (sufficient for analysis) -- **> 15 seconds**: Always kept (normal flight logs) - -**Examples:** -- 0.7s flight with 1500 frames → Skipped (too short) -- 2.2s flight with 4407 frames (2003 fps) → Kept (good data density) -- 10s flight with 8000 frames (800 fps) → Skipped (insufficient data density) -- 20s flight → Always kept (normal duration) - -Use `--force-export` to bypass filtering and export all logs regardless of duration or data quality. - -## Output - -### Console Statistics (Always Displayed) - -``` -Processing: flight_log.BBL - -Log 1 of 1, frames: 84235 -Firmware: Betaflight 4.5.2 (024f8e13d) STM32F7X2 -Board: AXFL AXISFLYINGF7PRO -Craft: Volador 5 - -Statistics -Looptime 125 avg -I frames 1316 -P frames 82845 -H frames 1 -G frames 833 -E frames 4 -S frames 6 -Frames 84235 -Data ver 2 -``` - -### Export Output (When Enabled) - -Additional output when export flags are used: -``` -Exported headers to: flight_log[.XX].headers.csv -Exported flight data to: flight_log[.XX].csv -Exported GPS data to: flight_log[.XX].gps.gpx # When --gpx used -Exported event data to: flight_log[.XX].event # When --event used -``` - -### Debug Output - -Debug mode adds frame data tables for detailed analysis: - -``` -=== FRAME DATA === - -I-frame data (25 frames): - Index Time(μs) Loop accSmooth accSmooth gyroADC[0] motor[0] motor[1] ... (40 more fields) - 0 0 4 0 0 -5 1270 1270 ... - 1 36147802 71168 -163 130 2289 1260 1277 ... - ... ... ... ... (18 frames skipped) - 23 36853826 73984 -332 -12 3512 1215 1210 ... - 24 36885919 74112 -430 26 3552 1205 1210 ... - -P-frame data (50 frames): - Index Time(μs) Loop accSmooth accSmooth gyroADC[0] motor[0] motor[1] ... (40 more fields) - 0 18446744073709551615 5 -11 9 27 632 637 ... - 1 18446744073709551615 6 -11 9 26 948 958 ... - ... ... ... ... (18 frames skipped) - 49 939855786 71193 -75 94 1504 854 841 ... -``` - -**Debug mode** provides detailed analysis including: -- File size and binary data inspection -- Field definitions and encoding details -- **Frame data tables** organized by type (I, P, S, G, E, H frames) -- Smart sampling: shows all frames ≤30, or first 5 + middle 5 + last 5 when >30 frames - -## Frame Support & Compatibility - -**Frame Types:** I, P, H, S, E, G frames -**Encoding:** All major BBL formats (SIGNED_VB, UNSIGNED_VB, NEG_14BIT, TAG8_8SVB, TAG2_3S32, TAG8_4S16) -**Predictors:** Reference-compliant implementation for P-frame decoding - -### Event Frame Support - -Event parsing uses the official Betaflight FlightLogEvent enum: -- **Type 0**: Sync beep (initialization) -- **Type 15**: Disarm event -- **Type 30**: Flight mode change -- **Type 255**: Log end marker -- **Additional types**: Autotune, inflight adjustment, logging resume events - -## Performance & Limitations - -**⚠️ Development Status**: This is work-in-progress software with the following limitations: -- Limited testing with GPS and Event frame processing -- May have compatibility issues with some specialized log formats -- Performance optimizations still in development -- API may change between versions - -**Current Capabilities:** -- Extensively tested with Betaflight and EmuFlight log files -- Memory-efficient streaming architecture for large files -- Processes files that may cause external decoders to fail -- Zero external dependencies - no blackbox_decode tools required - -## Development - -### Setting Up Development Environment - -To set up your development environment with proper formatting and pre-commit hooks: - -```bash -# Clone and setup -git clone -cd bbl_parser - -# Run setup script (optional but recommended) -chmod +x .github/setup-dev.sh -./.github/setup-dev.sh +# Useful options +./target/release/bbl_parser logs/*.BBL --output-dir ./output +./target/release/bbl_parser --force-export logs/*.BBL ``` -### Required Commands Before Committing - -**⚠️ IMPORTANT**: Always run these commands before committing to avoid CI failures: - -```bash -# 1. Format code (REQUIRED) -cargo fmt --all +## Output formats -# 2. Check formatting compliance -cargo fmt --all -- --check +- CSV: main flight data `[.XX].csv` and headers `[.XX].headers.csv` (field order matches blackbox_decode; time column is "time (us)") +- GPX: GPS track `[.XX].gps.gpx` +- Events: JSON Lines `[.XX].event` (CLI) -# 3. Check for clippy warnings (treated as errors) -cargo clippy --all-targets --all-features -- -D warnings +Filenames are clean for single-log files and numbered for multi-log files (e.g., `.01.csv`, `.02.csv`). -# 4. Run all tests -cargo test --verbose -cargo test --features=cli --verbose - -# 5. Verify release build -cargo build --release -``` - -### Pre-commit Hook (Recommended) - -Install the pre-commit hook to automatically format code: - -```bash -cp .github/pre-commit-hook.sh .git/hooks/pre-commit -chmod +x .git/hooks/pre-commit -``` +## Smart export filtering -This will automatically run `cargo fmt --all` before each commit and prevent commits with formatting issues. +To reduce noise from test arm/disarm logs: +- < 5s: skipped +- 5–15s: exported only if data density > 1500 fps +- > 15s: exported -### CI Requirements +Use `--force-export` to export everything. -The GitHub Actions CI enforces strict formatting and code quality: -- **Formatting**: Must pass `cargo fmt --all -- --check` -- **Linting**: Must pass `cargo clippy --all-targets --all-features -- -D warnings` -- **Testing**: Must pass all tests including CLI features -- **Building**: Must build successfully on Ubuntu, Windows, and macOS +## Documentation -## Dependencies - -- `clap` (v4.0+) - CLI parsing -- `glob` (v0.3) - File patterns -- `anyhow` (v1.0) - Error handling - -## Betaflight Firmware Compatibility - -**Flight Mode Flags, State Flags, and Failsafe Phases:** This parser outputs data that matches current Betaflight firmware specifications exactly. - -- **Flight Mode Flags**: Current `flightModeFlags_e` enum from Betaflight runtime_config.h - - Supports: ANGLE_MODE, HORIZON_MODE, MAG, BARO, GPS_HOLD, HEADFREE, PASSTHRU, FAILSAFE_MODE, GPS_RESCUE_MODE - - Output format: `"ANGLE_MODE|HORIZON_MODE"` (pipe-separated for CSV compatibility) - - Includes GPS_RESCUE_MODE flag (bit 11) from current firmware - -- **State Flags**: Current `stateFlags_t` enum from Betaflight runtime_config.h - - Supports: GPS_FIX_HOME, GPS_FIX, CALIBRATE_MAG, SMALL_ANGLE, FIXED_WING - - Output format: `"GPS_FIX_HOME|GPS_FIX"` (pipe-separated for CSV compatibility) - -- **Failsafe Phase**: Current `failsafePhase_e` enum from Betaflight failsafe.h - - Supports: IDLE, RX_LOSS_DETECTED, LANDING, LANDED, RX_LOSS_MONITORING, RX_LOSS_RECOVERED, GPS_RESCUE - - Includes phases 4-6 from current firmware - -**Reference Compatibility:** Verified against blackbox-tools and current Betaflight master branch. - -## Overview - -- [GOALS.md](./GOALS.md) -- [FRAMES.md](./FRAMES.md) -- [OVERVIEW.md](./OVERVIEW.md) - -## Contributing - -**⚠️ Development Project**: This is work-in-progress software. Contributions welcome with understanding that: -- APIs may change between versions -- Not all edge cases have been tested -- Performance optimizations are ongoing - -**Priority Areas:** Testing across firmware versions, file format compatibility, performance optimization +- Project overview: [OVERVIEW.md](./OVERVIEW.md) +- Frame details: [FRAMES.md](./FRAMES.md) +- Goals: [GOALS.md](./GOALS.md) +- Rust crate usage: [CRATE_USAGE.md](./CRATE_USAGE.md) +- Rust crate examples: [examples/README.md](./examples/README.md) ## License -This project is dual-licensed under: - -### Open Source License (AGPL-3.0-or-later) -Under the [GNU Affero General Public License v3.0 or later](LICENSE), you may use, modify, and distribute this software for both commercial and non-commercial purposes. However, this license imposes copyleft obligations that require any modified or redistributed versions of this work to also be licensed under the AGPL-3.0-or-later and make the source code available. **If you use this software to provide a network service, you must also make the source code available to users of that service.** See the [LICENSE](LICENSE) file for full terms. - -### Commercial License -For proprietary applications that need to use this project without AGPL-3.0-or-later's copyleft and network requirements, a separate commercial license is available as detailed in [LICENSE_COMMERCIAL](LICENSE_COMMERCIAL). Commercial use must abide by the terms in LICENSE_COMMERCIAL if you choose not to comply with AGPL-3.0-or-later. This option allows: - -Please contact the project maintainer for commercial licensing details. +Dual-licensed: +- Open source: [AGPL-3.0-or-later](./LICENSE) +- Commercial option: [LICENSE_COMMERCIAL](./LICENSE_COMMERCIAL) ## Acknowledgments -Based on [Betaflight Blackbox Log Viewer](https://github.com/betaflight/blackbox-log-viewer) and [blackbox-tools](https://github.com/betaflight/blackbox-tools) +Inspired by Betaflight's [blackbox-log-viewer](https://github.com/betaflight/blackbox-log-viewer) and [blackbox-tools](https://github.com/betaflight/blackbox-tools).