MSP protocol JSON definition file + derived documentation generation

[xznhj8129]

User description

This PR adds a master INAV-MSP definition file from which documentation, headers, APIs and more can be auto-generated.
Since INAV does not have foundational headers for MSP, i had to painstakingly parse the AIGen MSP reference wiki page and source code until all but a tiny handful of messages were correctly parsed along with their used enums and structs and a valid usable JSON file was generated. See https://github.com/xznhj8129/msp_documentation
The JSON file is meant as a base truth reference from which everything else can be derived so corrections can be made in one single location and propagated everywhere else.

Caveats:

• Some deprecated messages have not been fully fleshed out
• Some schema questions are open such as including the integer byte size, hex code or python struct symbol; and especially the handling of variable-structure payload messages; but the basic format is workable. Special edge cases are rare and easy to deal with.
• ctypes may have mistakes (source signed vs unsigned problem), but their reference was the very latest MSP reference page so the correctness level is the same and generally good.
• MSP_SET_VTX_CONFIG, MSP2_INAV_SET_GEOZONE_VERTEX, MSP2_COMMON_SET_SETTING, MSP2_SENSOR_HEADTRACKER have difficult payloads to parse and did not successfully parse automatically, but it will be a 5 minute job to correct them in the JSON file.
• Perhaps XML/XSD is better for this, but i'm more experienced with JSON, and it would be easy to convert anyway.

Again, once an error is corrected here, it corrects everywhere.

This will immensely simplify development of software that uses the MultiWii Serial Protocol and interfacing with INAV. This could allow to generate one-time a C header of message structs that can be validated, verified, included in the main source code and then used as main ground truth reference from which to generate JSON, C or Python code; simplifying development immensely.

---

PR Type

Enhancement, Documentation

---

Description

This description is generated by an AI tool. It may have inaccuracies

• Adds comprehensive MSP protocol documentation generation system with JSON-based message definitions

  • gen_msp_md.py: Generates Markdown documentation from MSP message definitions with strict code validation
  • gen_enum_md.py: Extracts and documents C enums from INAV source with preprocessor condition tracking
  • get_all_inav_enums_h.py: Consolidates all enums from INAV source directories into single header
  • manual_docs_fix.py: Provides manual documentation for complex variable-payload messages

• Includes master MSP definition JSON file (msp_messages.json) as single source of truth for protocol

• Generates auto-indexed Markdown reference (msp_ref.md) with request/reply payload tables and enum links

• Implements strict validation to ensure consistency between JSON definitions and MSPCodes enum

---

Diagram Walkthrough

flowchart LR
  A["INAV Source Code"] -->|extract| B["all_enums.h"]
  B -->|parse| C["gen_enum_md.py"]
  C -->|generate| D["inav_enums_ref.md"]
  E["msp_messages.json"] -->|parse| F["gen_msp_md.py"]
  F -->|validate| G["MSPCodes Enum"]
  F -->|generate| H["msp_ref.md"]
  I["manual_docs_fix.py"] -->|override| H

Loading

File Walkthrough

	Relevant files
Documentation	8 files gen_msp_md.py MSP message documentation generator with strict validation +539/-0  gen_enum_md.py C enum parser and Markdown documentation generator              +287/-0  manual_docs_fix.py Manual documentation overrides for complex messages            +56/-0    get_all_inav_enums_h.py Consolidates INAV source enums into single header                +78/-0    gen_docs.sh Orchestration script for documentation generation pipeline +13/-0    docs_v2_header.md Header template for generated MSP reference documentation +27/-0    msp_ref.md Generated MSP protocol reference with indexed messages      +4076/-0 inav_enums_ref.md Generated enumeration reference with condition tracking    +4598/-0
Configuration changes	1 files msp_messages.json Master MSP protocol message definitions in JSON format      +12855/-0

---

[qodo-code-review]

PR Compliance Guide 🔍

(Compliance updated until commit 56bb389)

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Insecure versioning Description: The script auto-increments a revision file based solely on md5sum comparison of msp_messages.json without locking or verification, which could be abused or cause unintended version bumps in concurrent or untrusted environments. gen_docs.sh [7-14] Referred Code actual="$(md5sum msp_messages.json | awk '{print $1}')" expected="$(awk '{print $1}' msp_messages.checksum)" if [[ "$actual" != "$expected" ]]; then n="$(cat rev)" echo $((n + 1)) > rev echo "File changed, incrementing revision" fi
	Information disclosure Description: The script prints all scanned file paths from the source tree which may leak sensitive project paths in logs or CI artifacts. get_all_inav_enums_h.py [94-99] Referred Code for fn in root.rglob('*'): print(fn) if fn.suffix in ('.c', '.h'): txt = fn.read_text(errors='ignore') ret = extract_enums(fn, txt) if ret: print(fn)
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Weak error handling: The shell script runs multiple commands without set -e/pipefail or handling non-zero exits, risking silent failures and inconsistent outputs. Referred Code echo "###########" echo get_all_inav_enums_h.py python get_all_inav_enums_h.py echo "###########" echo "msp_messages.json checksum" actual="$(md5sum msp_messages.json | awk '{print $1}')" expected="$(awk '{print $1}' msp_messages.checksum)" if [[ "$actual" != "$expected" ]]; then n="$(cat rev)" echo $((n + 1)) > rev echo "File changed, incrementing revision" fi echo "###########" echo gen_msp_md.py python gen_msp_md.py echo "###########" echo gen_enum_md.py ... (clipped 3 lines)
⚪	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: No audit logs: The new scripts and generators perform file operations and transformations without emitting any structured audit logs of actions, inputs, or outcomes. Referred Code echo "###########" echo get_all_inav_enums_h.py python get_all_inav_enums_h.py echo "###########" echo "msp_messages.json checksum" actual="$(md5sum msp_messages.json | awk '{print $1}')" expected="$(awk '{print $1}' msp_messages.checksum)" if [[ "$actual" != "$expected" ]]; then n="$(cat rev)" echo $((n + 1)) > rev echo "File changed, incrementing revision" fi echo "###########" echo gen_msp_md.py python gen_msp_md.py echo "###########" echo gen_enum_md.py ... (clipped 3 lines)
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Raw error output: The script prints file-not-found errors directly to stderr without distinguishing user-facing vs. internal logs, which may expose paths depending on usage context. Referred Code path = Path("all_enums.h") if not path.exists(): print(f"Error: {path} not found", file=sys.stderr) return 1 enums = parse_files([path]) md = render_markdown(enums) Path("inav_enums_ref.md").write_text(md, encoding="utf-8")
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Unvalidated inputs: The generator reads external JSON and local files without schema validation or try/except around file reads/parsing, which can cause unhandled exceptions or malformed output. Referred Code with open("docs_v2_header.md", "r", encoding="utf-8") as f: header = f.read() with open("format.md", "r", encoding="utf-8") as f: fmt = f.read() with open("msp_messages.checksum", "r", encoding="utf-8") as f: chksum = f.read().split(' ')[0] with open("rev", "r", encoding="utf-8") as f: rev = f.read() header = header.replace('<format>',fmt) header = header.replace('<file_rev>',rev) header = header.replace('<file_hash>',chksum) index_md = build_index(json_by_code) return header + "\n" + index_md + "\n" + "".join(sections) def main(): in_path = Path(sys.argv[1]) if len(sys.argv) >= 2 else Path("msp_messages.json") out_path = Path(sys.argv[2]) if len(sys.argv) >= 3 else Path("msp_ref.md") ... (clipped 7 lines)

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

---

Previous compliance checks

Compliance check up to commit 10a7416

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	Invert the data flow for documentation The documentation generation process should be inverted. Instead of relying on a manually-created JSON file, the scripts should parse the C source code as the definitive source of truth to generate documentation and other artifacts. Examples: docs/development/msp/gen_msp_md.py [26-276] class MSPCodes(enum.IntEnum): MSP_API_VERSION = 1 MSP_FC_VARIANT = 2 MSP_FC_VERSION = 3 MSP_BOARD_INFO = 4 MSP_BUILD_INFO = 5 MSP_INAV_PID = 6 MSP_SET_INAV_PID = 7 MSP_NAME = 10 MSP_SET_NAME = 11 ... (clipped 241 lines) docs/development/msp/msp_messages.json [1-12855] { "MSP_API_VERSION": { "code": 1, "hex": "0x1", "mspv": 1, "direction": 1, "request": null, "reply": { "size": 3, "struct": "<BBB", ... (clipped 12845 lines) Solution Walkthrough: Before: # docs/development/msp/gen_msp_md.py # 1. A massive, hardcoded enum is maintained inside the Python script # to validate against the JSON. class MSPCodes(enum.IntEnum): MSP_API_VERSION = 1 MSP_FC_VARIANT = 2 # ... hundreds of lines ... MSP2_BETAFLIGHT_BIND = 12288 def main(): # 2. The manually-created JSON is the primary input. in_path = Path("msp_messages.json") with in_path.open("r") as f: defs = json.load(f) # 3. The JSON is used to generate Markdown documentation. md = generate_markdown(defs) out_path.write_text(md) After: # gen_msp_definitions.py (New or modified script) def parse_c_source_for_msp_messages(source_files): # 1. Parse the actual INAV C source files to extract # MSP message definitions, structs, enums, and comments. # This becomes the true source of data. ... return extracted_definitions def main(): # 2. The C code is the primary input. source_files = find_inav_source_files() definitions = parse_c_source_for_msp_messages(source_files) # 3. (Optional) Generate the JSON file as an intermediate artifact. with open("msp_messages.json", "w") as f: json.dump(definitions, f) # 4. Generate Markdown from the extracted definitions. md = generate_markdown(definitions) Path("msp_ref.md").write_text(md) Suggestion importance[1-10]: 10 __ Why: This suggestion addresses a critical architectural flaw in the PR's design, as using a manual JSON file as the source of truth is fragile and prone to divergence from the actual C code, and proposes a much more robust and maintainable solution.	High
Possible issue	Prevent script from hanging in CI Modify the script to only pause for input when running in an interactive terminal, preventing it from hanging in automated environments like CI/CD. docs/development/msp/gen_docs.sh [9-13] echo "###########" echo gen_enum_md.py python gen_enum_md.py rm all_enums.h -read -n 1 -s -r -p "Press any key to continue" +# Pause only if running in an interactive terminal +if [ -t 0 ]; then + read -n 1 -s -r -p "Press any key to continue" +fi + Apply / Chat Suggestion importance[1-10]: 8 __ Why: This suggestion correctly identifies and fixes a significant issue where the script would hang in non-interactive environments like CI/CD, which is crucial for automation and build process stability.	Medium
	✅ Fix broken enum links in tables Suggestion Impact: The commit updated the navMcAltHoldThrottle_e links in both MSP_NAV_POSHOLD and MSP_SET_NAV_POSHOLD tables to point to the INAV wiki Enums-reference page, fixing the broken links. code diff: -| `mcAltHoldThrottleType` | `uint8_t` | 1 | [navMcAltHoldThrottle_e](https://github.com/iNavFlight/inav/wiki/inav_enums_ref.md#enum-navmcaltholdthrottle_e) | Enum `navMcAltHoldThrottle_e` Sets 'navConfigMutable()->mc.althold_throttle_type' | +| `mcAltHoldThrottleType` | `uint8_t` | 1 | [navMcAltHoldThrottle_e](https://github.com/iNavFlight/inav/wiki/Enums-reference#enum-navmcaltholdthrottle_e) | Enum `navMcAltHoldThrottle_e` Sets 'navConfigMutable()->mc.althold_throttle_type' | | `mcHoverThrottle` | `uint16_t` | 2 | PWM | Multirotor hover throttle (`currentBatteryProfile->nav.mc.hover_throttle`) | ## <a id="msp_set_nav_poshold"></a>`MSP_SET_NAV_POSHOLD (13 / 0xd)` **Description:** Sets navigation position hold and general manual/auto flight parameters. **Request Payload:** -| Field | C Type | Size (Bytes) | Units | Description | +|Field|C Type|Size (Bytes)|Units|Description| |---|---|---|---|---| | `userControlMode` | `uint8_t` | 1 | [ENUM_NAME](LINK_TO_ENUM) | Sets `navConfigMutable()->general.flags.user_control_mode | | `maxAutoSpeed` | `uint16_t` | 2 | cm/s | Sets `navConfigMutable()->general.max_auto_speed | @@ -442,7 +441,7 @@ | `maxManualSpeed` | `uint16_t` | 2 | cm/s | Sets `navConfigMutable()->general.max_manual_speed | | `maxManualClimbRate` | `uint16_t` | 2 | cm/s | Sets `fw.max_manual_climb_rate` or `mc.max_manual_climb_rate | | `mcMaxBankAngle` | `uint8_t` | 1 | degrees | Sets `navConfigMutable()->mc.max_bank_angle | -| `mcAltHoldThrottleType` | `uint8_t` | 1 | [navMcAltHoldThrottle_e](https://github.com/iNavFlight/inav/wiki/inav_enums_ref.md#enum-navmcaltholdthrottle_e) | Enum `navMcAltHoldThrottle_e` Sets 'navConfigMutable()->mc.althold_throttle_type' | +| `mcAltHoldThrottleType` | `uint8_t` | 1 | [navMcAltHoldThrottle_e](https://github.com/iNavFlight/inav/wiki/Enums-reference#enum-navmcaltholdthrottle_e) | Enum `navMcAltHoldThrottle_e` Sets 'navConfigMutable()->mc.althold_throttle_type' | | `mcHoverThrottle` | `uint16_t` | 2 | PWM | Sets `currentBatteryProfileMutable->nav.mc.hover_throttle | Fix the broken link for navMcAltHoldThrottle_e in the MSP_NAV_POSHOLD table to point to the correct enum documentation. docs/development/msp/msp_ref.md [430] -| `mcAltHoldThrottleType` | `uint8_t` | 1 | [navMcAltHoldThrottle_e](https://github.com/iNavFlight/inav/wiki/inav_enums_ref.md#enum-navmcaltholdthrottle_e) | Enum `navMcAltHoldThrottle_e` Sets 'navConfigMutable()->mc.althold_throttle_type' | +| `mcAltHoldThrottleType` | `uint8_t` | 1 | [navMcAltHoldThrottle_e](https://github.com/xznhj8129/msp_documentation/blob/master/docs/inav_enums_ref.md#navmcaltholdthrottle_e) | Enum `navMcAltHoldThrottle_e` Sets 'navConfigMutable()->mc.althold_throttle_type' | [Suggestion processed] Suggestion importance[1-10]: 5 __ Why: The suggestion correctly identifies and fixes a broken link, improving the documentation's usability by pointing to the correct reference mentioned in the document's header.	Low
General	Simplify integer literal parsing logic Simplify the integer parsing logic in is_plain_int_literal by replacing complex regex checks with a try-except block around int(t, 0). docs/development/msp/gen_enum_md.py [43-60] def is_plain_int_literal(expr: str) -> Optional[int]: """ Return int value if expr is a plain integer literal (dec/hex/bin/oct), otherwise None. Whitespace ok; no unary ops/casts/suffixes. """ t = expr.strip() if not t: return None - if re.fullmatch(r'0[xX][0-9A-Fa-f]+', t) or \ - re.fullmatch(r'0[bB][01]+', t) or \ - re.fullmatch(r'0[0-7]*', t) or \ - re.fullmatch(r'[1-9][0-9]*', t) or \ - t == '0': - try: - return int(t, 0) - except Exception: - return None - return None + try: + # Let Python's int() parser handle different bases (0x, 0b, 0o) + # and decimal numbers. It's more robust than regex for this. + return int(t, 0) + except ValueError: + return None Apply / Chat Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies that the regex-based integer validation is overly complex and can be robustly simplified using a try-except block with int(t, 0), improving code readability and maintainability.	Low
Update

[sensei-hacker]

Thanks for your work on this.

It looks like this PR doesn't change any functional files, only adds documentation-related files, so it should be safe.

I see a lot of stuff in ‎docs/development/msp/.gitignore‎ . Is that intended / needed?

[xznhj8129]

Thanks for your work on this.

It looks like this PR doesn't change any functional files, only adds documentation-related files, so it should be safe.

I see a lot of stuff in ‎docs/development/msp/.gitignore‎ . Is that intended / needed?

You're very welcome. And no, that .gitignore is just an auto-created vscode artifact of some sort, it's safe to get rid of