Pushed MowerMapTrace (accumulator + SVG renderer) on the same branch.

Addresses @edenhaus' review on home-assistant/core#169590 ("the creation of the map should go into the library"): moves the rendering and FIFO-cap accumulation out of the HA integration and into this lib. 9 unit tests added (tests/test_mower_trace.py). The HA Core PR is now a thin consumer.

Thanks for working on this. I have been analysing a separate GOAT MQTT capture (O1200 LiDAR Pro) containing both the static map and live mowing data. I think a few observations may be relevant to this PR.

1. index appears to represent chunks of one compressed stream

In my capture, payloads with the same (mid, batid, serial, type) and different index values must be assembled before LZMA decompression.

Only index=0 contains the LZMA header. Later indexes contain continuation bytes and cannot be decompressed independently.

The decoding flow that worked was:

key = (mid, batid, serial, type)

chunks[key][int(index)] = base64.b64decode(info)

compressed = b"".join(
    chunks[key][index]
    for index in sorted(chunks[key])
)

decoded = decompress_lzma(compressed)

infoSize appears to be the total byte length of the decompressed JSON, rather than the size of an individual compressed chunk. I use it as a completion check:

len(decoded) == infoSize

This also seems consistent with the sample in issue #1376, where index=0 begins with the LZMA header and index=1 appears to continue the same compressed stream.

2. serial may not be a monotonic trace/cycle identifier

I would be careful about using serial as MapTraceEvent.start.

The logs in #1376 contain several different batid values with the same serial=2. My own captures also suggest that serial identifies a dataset or layer and may be reused across multiple batches.

At least (mid, batid, serial, type) should probably be preserved when assembling or resetting state.

3. Group and segment boundaries should probably be preserved

The current implementation drops the leading "0" and concatenates all segments from all groups into one polyline.

That may create artificial lines between otherwise independent segments:

end of group 5 segment 1
    -> start of group 5 segment 2
    -> start of group 6 segment 1

In other GOAT map payloads I have decoded, group IDs, subtypes and record IDs carry important semantic information. I would therefore preserve the original structure, for example:

[
    {
        "group_id": "5",
        "segments": [
            [(x1, y1), (x2, y2)],
            [(x3, y3), (x4, y4)],
        ],
    },
]

A flattened compatibility representation could still be generated later, but the original group and path-break information would not be lost.

The leading "0" may be a segment-start/path-break marker rather than an ordinary point. Dropping it as a coordinate makes sense, but the segment boundary itself should probably remain represented.

4. GOAT devices also expose a complete static map

My capture confirms the maintainer's observation that GOAT devices have a full map in addition to the trajectory data.

For the device I tested, the relevant MQTT data is split approximately as follows:

onMI
    main map boundary

onArI, layer 1
    area/zone boundaries

getAreaSet
    area IDs, names, adjacency and reference metadata

onMapTrack
    live mowing plan, remaining rows and working contours

onPos
    live mower position

The static onMI and onArI geometry is encoded as a start coordinate followed by an RLE-compressed eight-direction path:

DIRECTIONS = {
    "1": ( 1,  0),
    "2": ( 1, -1),
    "3": ( 0, -1),
    "4": (-1, -1),
    "5": (-1,  0),
    "6": (-1,  1),
    "7": ( 0,  1),
    "8": ( 1,  1),
}

One decoded direction step is 50 coordinate units in my capture.

The area records are not always stored at their final main-map position. I was able to place them without hardcoded offsets by:

1. decoding the main-map and area direction sequences;
2. finding their longest shared contiguous direction sequence;
3. using the corresponding points as anchors;
4. translating the complete area polygon into the main-map coordinate system.

offset_x = main_anchor_x - area_anchor_x
offset_y = main_anchor_y - area_anchor_y

The matched boundaries were then verified point by point. For all three zones in my sample, every point in the selected shared boundary matched exactly after translation.

This may provide a path towards implementing the regular map capability for GOAT devices rather than only rendering a cumulative trace.

5. Static maps and live tracks should remain separate concepts

The captures suggest that these are distinct datasets:

• static main-map boundary;
• persistent area polygons;
• live planned mowing rows;
• completed or remaining work contours;
• robot-position updates.

Flattening all of these into one cumulative trajectory may make an initial image possible, but it will make it difficult to reproduce what the Ecovacs app displays.

I can provide sanitised MQTT samples, decoded JSON files and the experimental Python viewer used to verify the map and zone registration if that would be useful.

Thanks for the extremely detailed review — these observations land squarely on the limits of the current PR. Your captures of an O1200 LiDAR Pro reach further than the single A1600 firmware sample I had, and 1–3 + 5 are clearly blocking issues for a correct implementation.

Here is what I plan to revise on this PR, plus what I think belongs in a follow-up:

1. Chunk assembly before LZMA decompression — will fix

You are right. The current handler treats every index as a standalone compressed blob and only succeeds because the single-chunk A1600 capture happened to fit in index=0. The fix is to buffer per (mid, batid, serial, type) and only decompress once the concatenated payload reaches infoSize. I will refactor along the lines of:

key = (mid, batid, serial, type_)
chunks[key][int(index)] = base64.b64decode(info)

if sum(len(c) for c in chunks[key].values()) >= info_size:
    compressed = b"".join(chunks[key][i] for i in sorted(chunks[key]))
    decoded = decompress_lzma(compressed)
    assert len(decoded) == info_size
    chunks.pop(key)

with a bounded buffer (drop incomplete keys after N seconds or M kB) so a missed final chunk doesn't leak memory across mowing cycles.

2. serial is not monotonic — will fix

Confirmed against the #1376 sample (same serial=2 across different batid values). Using it as MapTraceEvent.start was incorrect; it bypassed the Map helper's reset only by accident. I'll switch the assembly key to (mid, batid, serial, type) as you suggest, and stop overloading start with semantics it doesn't have.

3. Group and segment boundaries — will preserve

Agreed, flattening was a deliberate shortcut to land something usable in HA before the design was settled, but it conflates information that may matter (path breaks, distinct contours, per-group semantics). I'll switch the event payload to:

[
  {"group_id": "5", "segments": [[(x1, y1), ...], [(x3, y3), ...]]},
  ...
]

and keep the flat string as an optional compatibility projection, generated downstream if a consumer still needs it. The leading "0" becomes an explicit segment-start marker rather than being silently dropped.

4. Static map via onMI / onArI / getAreaSet — follow-up PR

This is genuinely new scope (RLE-encoded eight-direction path, area registration via shared-boundary anchor matching) and I'd rather not bundle it with the live trace fix to keep this PR reviewable. I will open a separate PR once #1567 lands, and would very much welcome the sanitised samples + your experimental viewer for that work — it would let me validate the anchor-matching approach against a known-good reference instead of guessing.

5. Keep static maps and live tracks as distinct concepts — agreed

This is the design principle that holds the other four fixes together. I'll make sure the refactored event keeps onMapTrack (live mowing plan), onPos (live position), and the eventual onMI/onArI static geometry on separate channels rather than concatenating them into one trajectory.

Samples

If you can share the sanitised MQTT captures + decoded JSON + your Python viewer, that would be hugely helpful — both for landing 1-3 correctly here and for scoping the static-map PR. Happy to receive them as a gist, a PR to my fork, or any other channel that suits you.

I'll push a revised commit on this branch addressing 1–3 + 5 over the next few days.

Codecov Report

❌ Patch coverage is 97.87234% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.13%. Comparing base (7fb6b38) to head (6f1a5b7).
⚠️ Report is 10 commits behind head on dev.

@@            Coverage Diff             @@
##              dev    #1567      +/-   ##
==========================================
+ Coverage   95.02%   95.13%   +0.11%     
==========================================
  Files         159      161       +2     
  Lines        6234     6439     +205     
  Branches      353      376      +23     
==========================================
+ Hits         5924     6126     +202     
- Misses        248      249       +1     
- Partials       62       64       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Merging this PR will not alter performance

✅ 226 untouched benchmarks

_{Comparing Beennnn:feat/parse-mower-getmaptrace-fw-1.15 (6f1a5b7) with dev (2a7b432)}

If you can share the sanitised MQTT captures + decoded JSON + your Python viewer, that would be hugely helpful — both for landing 1-3 correctly here and for scoping the static-map PR. Happy to receive them as a gist, a PR to my fork, or any other channel that suits you.

Thanks for the response. I have prepared a sanitised package containing the material I have collected and the tools I have used so far:

The package includes:

• sanitised MQTT captures for the relevant static-map and live-map topics, including onMI, onArI, onMapTrack, onPos and related request/response messages;
• the decoded JSON output from the compressed map payloads;
• a chunk-aware Base64/LZMA decoding example;
• a Python viewer for reconstructing the static main map and registering the area boundaries;
• an experimental viewer for the live MapTrack data, planned mowing rows and robot position;
• normalised JSON exports that preserve groups, record IDs, batch IDs and segment boundaries;
• notes describing the observed formats, assumptions, confirmed findings and unresolved questions;
• a sanitisation report describing which identifiers and credentials were removed or replaced.

The most important findings in the package are:

1. Payload chunks with the same (mid, batid, serial, type) appear to be parts of one compressed LZMA stream and must be concatenated in index order before decompression.

2. infoSize appears to represent the size of the decompressed JSON data.

3. serial does not appear to be a unique batch or mowing-session identifier by itself. Different batid values may reuse the same serial.

4. Group IDs, subtypes, record IDs and segment boundaries should be preserved. Flattening everything into one polyline may create artificial connections between unrelated paths.

5. The static main map and area boundaries are represented as run-length-encoded eight-direction paths.

6. Area boundaries can be registered against the main map without hardcoded offsets by finding the longest shared contiguous direction sequence and using the corresponding points as anchors.

7. The live onMapTrack data appears to contain separate records for planned rows, working contours and incremental updates. This part is still experimental and should not yet be considered a complete protocol specification.

A small disclaimer: I am not a software developer and do not have a strong background in protocol reverse engineering. I have worked through this using the limited technical knowledge I have, a lot of testing and comparison, and assistance from AI tools. The viewers and documentation should therefore be treated as reference material and experimental findings rather than production-ready code.

I am happy to collect more captures, test hypotheses or inspect specific app actions. It would help me greatly if requests could be made as clearly and specifically as possible, for example:

• perform one named action in the app;
• capture the MQTT traffic before, during and after that action;
• provide a capture while the mower is in a particular state;
• compare two settings while changing only one value;
• verify a specific field or message type.

With a clear description of the required action and the data you need, I will do my best to provide additional sanitised captures and results.

goat_map_mqtt_capture_sanitized.zip

Thanks for the sanitised package — this is genuinely more than I was hoping for, and it lines up cleanly with what's already in this PR after the refactor. Cross-checking your FORMAT_NOTES.md against the current diff:

§1 — LZMA-alone header repair. The insert-four-zeros-at-byte-8 step is already what the existing Rust helper decompress_base64_data does (src/util.rs:36-38), and this PR reuses it as-is for the chunked path.

§2 — Chunk assembly with (mid, batid, serial, type). Same key here (_CHUNK_BUFFER: dict[tuple[str, str, str, str], dict[int, bytes]]), with an empty string when type is absent, and reassembly sorted by integer index before decompression. Your finding §3 — "serial isn't a unique batch id by itself" — is precisely why batid is part of the key rather than a bare serial lookup.

§7 — preserve structure. The refactor now emits MowerMapTraceEvent(groups=list[MowerMapTraceGroup]), each with its own group_id and segments: list[MowerMapTraceSegment], so group/segment breaks and record IDs survive. The flat data field is derived once for legacy consumers, not the source of truth. Original raw record text isn't kept — happy to add it if you'd find it useful downstream.

§3, §4, §5, §6, §8 — static main map, onArI layer-1 zones, getAreaSet, onMapTrack full/patch, deebotPos/chargePos semantics. These are exactly the surface I was hoping to scope for the follow-up static-map PR. The RLE 8-direction paths + longest-shared-sequence anchor alignment, the ["1","1"] vs ["1","2"] full/patch discrimination on onMapTrack, the chargePos.invalid flag on onPos — all of that would land there rather than trying to bolt it onto onMapTrace.

Two small pinpoint questions to make sure I map them correctly:

1. In §6, is the "route ID" you key state by exactly the triple (record_type, subtype, record_id) from 2;1;99;… — i.e. (2, 1, 99) — or is one of those a per-batch counter that shouldn't be part of the identity?
2. In §4, when the longest shared direction sequence returns zero, do you drop the zone or fall back to local_start_x,local_start_y as-is? A recipe for the degenerate case would save me a wrong assumption later.

If you're OK with it, I'd like to open the follow-up PR referencing docs/FORMAT_NOTES.md from your package (with attribution) as the interpretation source. Nothing you shared changes the shape of this PR — the refactor already reflects it — so I don't think #1567 needs another push on my side unless you spot something the diff still gets wrong.

Thanks, that all sounds good to me, and I am happy for you to reference docs/FORMAT_NOTES.md from the package with attribution.

Also good to hear that the current refactor already matches the chunking and structure-preserving parts of the findings. I do not currently see anything in #1567 that obviously contradicts the captures I shared.

Regarding your two questions:

1. Route identity in onMapTrack

For the live onMapTrack state, the key I used is the full triple:

(record_type, subtype, record_id)

So for:
2;1;99;...
I treated the identity as:
("2", "1", "99")

In my capture, this was important because the same record_id value may have different meaning depending on the first two fields.

My current interpretation is:

the first field is a high-level work/status group;
the second field is a subtype, for example rows vs contour-like geometry;
the third field is the record ID within that namespace.

For example, the active row was repeatedly updated as:
2;1;99;...
The later onMapTrack patch messages did not append a new route. They replaced the current state for the same key ("2", "1", "99").

So my current recipe is:

key = (parts[0], parts[1], parts[2])
records[key] = parsed_record

For a full snapshot, I would replace the whole current onMapTrack state. For a patch/update, I would update records by that triple key.

I would not treat 99 alone as the route identity.

I also would not yet assume that any of the three fields is globally unique across different batches or map sessions. The triple is only the safest observed identity inside the current reconstructed onMapTrack state.

Keeping the original raw record text would be useful downstream if it is not too much trouble. It makes it much easier to re-check assumptions later, especially while the meaning of the first two fields is still being reverse engineered.

2. Longest shared sequence returns zero

For static area alignment, I would not fall back to local_start_x, local_start_y as if it were already in the main-map coordinate system.

In my sample, the area start coordinates were local to the area geometry and were not always the final global placement. The correct placement came from the shared-boundary anchor, not from the local start coordinate.

So my suggested degenerate-case recipe is:

Decode the area geometry and keep it in the output as raw/local geometry.
Try to find the longest shared contiguous direction sequence against the main map.
If a sufficiently long match is found, compute the translation from the two anchor points and mark the area as registered.
If the match length is zero, or below a confidence threshold, do not place the area onto the main map as a confirmed polygon.
Mark it as unregistered / local-only / low-confidence in the parsed result.

Pseudo-logic:

match = longest_shared_sequence(main_tokens, area_tokens)

if match.size >= MIN_SHARED_STEPS:
    offset = main_points[match.main_index] - area_points[match.area_index]
    registered_points = translate(area_points, offset)
    registered = True
else:
    registered_points = None
    local_points = area_points
    registered = False

For the sample I shared, the matches were large:

area 1: 399 shared direction steps
area 2: 859 shared direction steps
area 4: 583 shared direction steps

and every point along the selected shared boundary matched exactly after translation.

I would therefore treat a zero-length match as “we do not know how to place this area yet”, rather than assuming the local start coordinate is a valid global anchor.

A later fallback might be possible using getAreaSet reference coordinates, bounding boxes, adjacency, or another contour layer, but I have not verified that yet. In my capture, the getAreaSet reference coordinates did not behave like direct polygon origins.

One extra robustness point: I have only verified the direct same-direction boundary match in my own capture. A future parser may also want to try the reversed boundary direction, since another firmware could theoretically encode the same shared boundary in the opposite traversal direction. But if both direct and reversed matching fail, I would keep the area unregistered rather than placing it incorrectly.

Thanks again for picking this up. I agree that #1567 does not need to absorb the static-map logic. Keeping this PR focused on the structured mower onMapTrace parser and then doing onMI / onArI / getAreaSet / onMapTrack in a follow-up static-map PR sounds like the right split to me.

And please remember I am still learning while working on this, so please excuse any mistakes, wrong assumptions or misunderstandings in my notes or terminology.

Thanks, that's exactly the recipes I needed.

I just pushed d521cca which adds raw: str | None = None to MowerMapTraceSegment and populates it with the original firmware segment string during parsing. So the leading-marker + x,y split assumptions can be re-verified downstream without going back to the compressed stream. Kept as None by default so old constructor call-sites don't need touching, and default-frozen dataclass equality still works.

On your two recipes — both go verbatim into the follow-up static-map PR, no reinterpretation needed on my side:

Route identity — triple (record_type, subtype, record_id), replacement semantics on both full snapshots and patches, no per-field uniqueness assumption across batches. That maps cleanly to a dict[tuple[str, str, str], Record] keyed on the parsed triple, updated in place. Your point that the same record_id value carries different meaning under different (record_type, subtype) pairs is exactly the kind of thing the raw field will help sanity-check as more captures come in.

Zero-shared-sequence fallback — no coercion of local coordinates into the main-map frame. I'll model each area as {registered_points: list | None, local_points: list, registered: bool, shared_steps: int} and require shared_steps >= MIN_SHARED_STEPS (starting at some conservative floor like 100, tunable against your capture where the smallest confirmed match was 399) before setting registered=True. The reversed-boundary attempt goes as a second pass before giving up. Everything below threshold ends up as registered=False with the local geometry preserved — the UI/consumer decides whether to render it at all.

I'll also lean on docs/FORMAT_NOTES.md from your package as the reference source in that follow-up PR, with attribution.

And nothing to excuse — the notes are precise and the failure modes you called out (per-batch identity, degenerate anchor, reversed traversal) are the ones that would have bitten a first-pass implementation. Much easier to design the follow-up against them than to discover them after the fact.