Log Analysis Introduced with backend_log_extraction.py by OmkarSarkar204 · Pull Request #1748 · ArduPilot/MethodicConfigurator

OmkarSarkar204 · 2026-06-19T06:18:35Z

Description

Pr Introducing the Log_Analysis module for Ardupilot log, which will be used by and with AMC

Checklist

Run pre-commit checks locally
Verified by a human programmer
All commits are signed off (use git commit --signoff)
Code follows our coding standards
[] Documentation updated if needed
No breaking changes or properly documented

Testing

Describe how you tested these changes:

Unit tests pass( for existing files not real files)
Integration tests pass
Manual testing performed
Tested on flight controller hardware

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Introduces a new log_analysis module to parse ArduPilot .bin logs, extracting parameter values/defaults, firmware version info, message counts, and frame type, and reuses that logic from extract_param_defaults.py.

Changes:

Added log_analysis/backend_log_extraction.py with log parsing + extraction helpers (params, firmware version, frame type).
Added package initializer log_analysis/__init__.py.
Refactored extract_param_defaults.py to reuse shared validation/version-parsing helpers from the new module.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.

File	Description
ardupilot_methodic_configurator/log_analysis/backend_log_extraction.py	New backend log extraction implementation (open/close log, parse messages, extract params/firmware/frame type).
ardupilot_methodic_configurator/log_analysis/init.py	Initializes the new `log_analysis` package with a module docstring.
ardupilot_methodic_configurator/extract_param_defaults.py	Replaces in-file regex/version parsing with imports from `backend_log_extraction`.

OmkarSarkar204 · 2026-06-19T07:45:14Z

The codespell was failing

codespell................................................................Failed
- hook id: codespell
- exit code: 65

ardupilot_methodic_configurator/data_model_configuration_step.py:351: SER ==> SET

So i just supressed it for now

coveralls · 2026-06-19T10:33:43Z

Coverage Report for CI Build 28046795869

Coverage decreased (-0.7%) to 93.678%

Details

Coverage decreased (-0.7%) from the base build.
Patch coverage: No coverable lines changed in this PR.
6 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

6 previously-covered lines in 1 file lost coverage.

File	Lines Losing Coverage	Coverage
extract_param_defaults.py	6	95.88%

Coverage Stats


Relevant Lines:	13651
Covered Lines:	12788
Line Coverage:	93.68%
Relevant Branches:	2096
Covered Branches:	2014
Branch Coverage:	96.09%
Branches in Coverage %:	No
Coverage Strength:	2.8 hits per line

💛 - Coveralls

amilcarlucas · 2026-06-20T14:33:39Z

Thanks for this. I reviewed the code and did some improvements.

OmkarSarkar204 · 2026-06-20T14:47:44Z

Thanks for this. I reviewed the code and did some improvements.

I was thinking of adding the test once ive finished the basic signals, but Thank You , Im working on the PM, IMU , VIBE and ERR messages and will try to do it by today midnight. :)

OmkarSarkar204 · 2026-06-22T06:44:13Z

Extracted till now
PARM ,VER, MSG (version fallback), BAT, PM, IMU, VIBE, GPS, ERR, ARM, MODE

OmkarSarkar204 · 2026-06-22T09:03:19Z

I'll Add more regression test after the quality_checker

amilcarlucas · 2026-06-23T10:11:11Z

All test pass, this looks good. But the ArduPilot .bin log format is self contained, meaning the the msages define themselves in the file first, before they get stored, meaning that the parser can be built that parses all messages, without knowing what is inside.
This way a parser can be written once, and never touched again. I think pymavlink does that.

amilcarlucas · 2026-06-23T14:05:18Z

The .bin files are "self-contained" they contain all of the information required to decode the file inside the file itself.
The FMT messages in the beginning of the file do exactly that.

OmkarSarkar204 · 2026-06-23T18:15:12Z

I'm just not sure abou the return type here
https://github.com/ArduPilot/MethodicConfigurator/pull/1748/changes#diff-0b403f20a6dd87694090b93f2752e5d38c3134ed147314c9b3155226d2171963R49

amilcarlucas · 2026-06-24T07:21:25Z

Sorry, Can you give a concrete file and line number?

This is what I meant: https://github.com/averyanalex/ardupilot-binlog but it is written in Rust. Should be straigtforward to convert it to python using AI.

OmkarSarkar204 · 2026-06-24T07:46:47Z

Sorry, Can you give a concrete file and line number?

The return type for

def parse_log(logfile: str) -> Any:  # noqa: ANN401
    """
    Open a log and yield each DataFlash message.

… param_default to remove code duplication. Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

OmkarSarkar204 · 2026-06-26T17:52:45Z

+        if not schema.fields:
+            issues.append("Missing field definitions")
+        if not schema.format:
+            issues.append("Missing format string")
+        if schema.length <= 0:
+            issues.append("Invalid message length")
+        if schema.units and len(schema.units) != len(schema.fields):
+            issues.append("Unit count mismatch")
+        if schema.multipliers and len(schema.multipliers) != len(schema.fields):
+            issues.append("Multiplier count mismatch")
+


Im sceptical is these checks are necessary or not ,as pymavlink already handles these i think, but if by any chance a log gets malformed midway, it will be usefull maybe?

Keep it around while you test multiple logs.

Mark it for later removal, if it duplicates pymavlink functionality

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

amilcarlucas

Looks good. By plugins I meant: ARCHITECTURE_battery_monitor.md and ARCHITECTURE_motor_test.md

amilcarlucas · 2026-06-27T10:36:07Z

@@ -0,0 +1,62 @@
+{


This should be done the other way around. The configuration_steps_*.json file should define something like:

"10_battery_monitor.param": { "why": "The battery monitor must be configured to match the hardware connection type so that the autopilot correctly reads battery voltage and current", "why_now": "After the battery voltage and failsafe thresholds are set, the monitor hardware connection parameters can be configured", "blog_text": "Configure the battery monitor hardware connection type, protocol and calibration values", "blog_url": "https://ardupilot.github.io/MethodicConfigurator/TUNING_GUIDE_ArduCopter#65-configure-the-primary-battery-monitor", "wiki_text": "Battery Monitors (aka Power Monitors/Modules) - BATT_* parameters", "wiki_url": "https://ardupilot.org/copter/docs/common-powermodule-landingpage.html", "external_tool_text": "", "external_tool_url": "", "mandatory_text": "100% mandatory (0% optional)", "component": "Battery Monitor", "auto_changed_by": "", "derived_parameters": { "BATT_MONITOR": { "New Value": "vehicle_components['Battery Monitor']['FC Connection']['Protocol']", "Change Reason": "Selected in component editor window" }, "BATT_I2C_BUS": { "if": "vehicle_components['Battery Monitor']['FC Connection']['Type'] in ['I2C1', 'I2C2', 'I2C3', 'I2C4']", "New Value": "1 if vehicle_components['Battery Monitor']['FC Connection']['Type'] == 'I2C2' else 2 if vehicle_components['Battery Monitor']['FC Connection']['Type'] == 'I2C3' else 3 if vehicle_components['Battery Monitor']['FC Connection']['Type'] == 'I2C4' else 0", "Change Reason": "Selected in component editor window" } }, "delete_parameters": { "BATT_I2C_BUS": { "if": "vehicle_components['Battery Monitor']['FC Connection']['Type'] not in ['I2C1', 'I2C2', 'I2C3', 'I2C4']" }, "BATT_AMP_OFFSET": { "if": "vehicle_components['Battery Monitor']['FC Connection']['Type'] != 'Analog' or vehicle_components['Battery Monitor']['FC Connection']['Protocol'] in ['Analog Voltage Only', 'FuelLevelAnalog']" }, "BATT_AMP_PERVLT": { "if": "vehicle_components['Battery Monitor']['FC Connection']['Type'] != 'Analog' or vehicle_components['Battery Monitor']['FC Connection']['Protocol'] in ['Analog Voltage Only', 'FuelLevelAnalog']" }, "BATT_VOLT_MULT": { "if": "vehicle_components['Battery Monitor']['FC Connection']['Type'] != 'Analog'" } }, "old_filenames": [], "plugin": { "name": "battery_monitor", "placement": "left" }, "related_bin_log_messages": { "BAT": { "name": "Battery" } } },

This way we can support multiple vehicles, and reduce maintenance effort.

Sure that was just temporary, ill do in configuration_steps_copter.json only , and ill start by the 100% mandatory mesasges. But im doing the categorisation based on some docs so it might be wrong. Ill update soon

amilcarlucas · 2026-06-27T10:43:21Z

+        if not schema.fields:
+            issues.append("Missing field definitions")
+        if not schema.format:
+            issues.append("Missing format string")
+        if schema.length <= 0:
+            issues.append("Invalid message length")
+        if schema.units and len(schema.units) != len(schema.fields):
+            issues.append("Unit count mismatch")
+        if schema.multipliers and len(schema.multipliers) != len(schema.fields):
+            issues.append("Multiplier count mismatch")
+


Keep it around while you test multiple logs.

Mark it for later removal, if it duplicates pymavlink functionality

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

Copilot AI review requested due to automatic review settings June 19, 2026 06:18

OmkarSarkar204 requested a review from amilcarlucas as a code owner June 19, 2026 06:18

OmkarSarkar204 marked this pull request as draft June 19, 2026 06:18

Copilot AI reviewed Jun 19, 2026

View reviewed changes

Copilot started reviewing on behalf of OmkarSarkar204 June 19, 2026 07:01 View session

OmkarSarkar204 force-pushed the log_analysis_feats branch 5 times, most recently from f4b3beb to b1e5229 Compare June 19, 2026 07:44

OmkarSarkar204 force-pushed the log_analysis_feats branch from c6b4406 to ace0a54 Compare June 19, 2026 07:51

amilcarlucas requested changes Jun 19, 2026

View reviewed changes

Comment thread ardupilot_methodic_configurator/log_analysis/backend_log_extraction.py Outdated

OmkarSarkar204 force-pushed the log_analysis_feats branch 2 times, most recently from bdf514b to 4239dca Compare June 19, 2026 14:39

OmkarSarkar204 changed the title ~~Log Analysis Introduced with ardupilot_log_extraction.py~~ Log Analysis Introduced with backend_log_extraction.py Jun 20, 2026

OmkarSarkar204 force-pushed the log_analysis_feats branch 4 times, most recently from f5a17b7 to 9c9ba7c Compare June 20, 2026 05:32

amilcarlucas force-pushed the log_analysis_feats branch from 9c9ba7c to d056d57 Compare June 20, 2026 14:32

amilcarlucas force-pushed the log_analysis_feats branch from d056d57 to 87d50fa Compare June 20, 2026 14:41

OmkarSarkar204 force-pushed the log_analysis_feats branch from 1e4e8b6 to 2003f98 Compare June 22, 2026 06:40

OmkarSarkar204 force-pushed the log_analysis_feats branch 3 times, most recently from e32105f to 871c7f6 Compare June 22, 2026 07:37

OmkarSarkar204 marked this pull request as ready for review June 22, 2026 09:02

OmkarSarkar204 requested a review from amilcarlucas June 22, 2026 09:02

OmkarSarkar204 force-pushed the log_analysis_feats branch 2 times, most recently from 05318c9 to 5861ec9 Compare June 23, 2026 18:08

OmkarSarkar204 and others added 6 commits June 26, 2026 23:07

feat: added backend_log_extraction.py for log_analysis and refactored…

b08d5e5

… param_default to remove code duplication. Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

fix: copilot suggestions

dc6ee05

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

feat: added battery extraction

577c7a7

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

test(backend log extraction): Added regression tests

b22e5c7

feat: added PM, IMU, VIBE, GPS, ERR, MODE, ARM messages

3289a61

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

refactor: separated parsing and extraction

7d78ccc

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

OmkarSarkar204 force-pushed the log_analysis_feats branch 3 times, most recently from e3a0ed6 to 5aa31a3 Compare June 26, 2026 17:48

OmkarSarkar204 commented Jun 26, 2026

View reviewed changes

OmkarSarkar204 force-pushed the log_analysis_feats branch 2 times, most recently from 947215e to 5a75c29 Compare June 27, 2026 03:31

refactor: log parsing and validation introduced

75352a8

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

OmkarSarkar204 force-pushed the log_analysis_feats branch from 5a75c29 to 75352a8 Compare June 27, 2026 03:52

amilcarlucas requested changes Jun 27, 2026

View reviewed changes

OmkarSarkar204 force-pushed the log_analysis_feats branch from 582b936 to 47ae7f1 Compare June 28, 2026 17:10

refactor: added basic checks in json

c70df12

Signed-off-by: Omkar Sarkar <omkarsarkar24@gmail.com>

OmkarSarkar204 force-pushed the log_analysis_feats branch from 47ae7f1 to c70df12 Compare June 28, 2026 17:13

OmkarSarkar204 requested a review from amilcarlucas June 28, 2026 17:16

Uh oh!

Conversation

OmkarSarkar204 commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OmkarSarkar204 commented Jun 19, 2026

Uh oh!

Uh oh!

coveralls commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for CI Build 28046795869

Coverage decreased (-0.7%) to 93.678%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

amilcarlucas commented Jun 20, 2026

Uh oh!

OmkarSarkar204 commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OmkarSarkar204 commented Jun 22, 2026

Uh oh!

OmkarSarkar204 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amilcarlucas commented Jun 23, 2026

Uh oh!

amilcarlucas commented Jun 23, 2026

Uh oh!

OmkarSarkar204 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amilcarlucas commented Jun 24, 2026

Uh oh!

OmkarSarkar204 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OmkarSarkar204 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

amilcarlucas Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

amilcarlucas left a comment

Choose a reason for hiding this comment

Uh oh!

amilcarlucas Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

OmkarSarkar204 Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amilcarlucas Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

OmkarSarkar204 commented Jun 19, 2026 •

edited

Loading

coveralls commented Jun 19, 2026 •

edited

Loading

OmkarSarkar204 commented Jun 20, 2026 •

edited

Loading

OmkarSarkar204 commented Jun 22, 2026 •

edited

Loading

OmkarSarkar204 commented Jun 23, 2026 •

edited

Loading

OmkarSarkar204 commented Jun 24, 2026 •

edited

Loading

OmkarSarkar204 Jun 27, 2026 •

edited

Loading