Add MacDive SQLite import (Milestone 3 of 4)#256
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new MacDive SQLite import pathway to the universal import system, aiming to provide a “rich metadata” migration path (vs. UDDF for profile samples) and improving cross-format deduplication via source_uuid.
Changes:
- Introduces
ImportFormat.macdiveSqlitewith detector + parser/reader/mapper wiring for MacDive Core Data SQLite files. - Adds a binary plist (
bplist00) decoder plus golden tests/fixtures to support MacDive Core Data BLOB decoding needs (notablyZTIMEZONE). - Enhances dive duplicate detection with a first-pass exact match on
source_uuid, backed by a new bulk repository query.
Reviewed changes
Copilot reviewed 31 out of 35 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| test/fixtures/macdive_sqlite/build_synthetic_db.dart | Builds a minimal MacDive-shaped SQLite DB fixture for unit/integration tests. |
| test/fixtures/macdive_sqlite/bplist_samples/small_dict.bplist | Golden bplist fixture (dict) for decoder tests. |
| test/fixtures/macdive_sqlite/bplist_samples/sample_array.bplist | Golden bplist fixture (array) for decoder tests. |
| test/fixtures/macdive_sqlite/bplist_samples/nested.bplist | Golden bplist fixture (nested structures) for decoder tests. |
| test/fixtures/macdive_sqlite/bplist_samples/macdive_ztimezone.bplist | Real-sample MacDive ZTIMEZONE bplist fixture for decoder coverage. |
| test/features/weather/data/repositories/weather_repository_test.mocks.dart | Updates mocks for new DiveRepository.getSourceUuidByDiveId. |
| test/features/universal_import/presentation/providers/universal_import_notifier_test.dart | Adds tests for MacDive SQLite detection and parser output (synthetic DB). |
| test/features/universal_import/presentation/providers/import_consolidation_service_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/universal_import/data/services/macdive_dive_mapper_test.dart | Validates MacDive SQLite mapping output shape from synthetic DB. |
| test/features/universal_import/data/services/macdive_db_reader_test.dart | Tests MacDive SQLite schema detection + row-graph reading. |
| test/features/universal_import/data/services/import_duplicate_checker_test.dart | Adds coverage for source_uuid-based dive duplicate detection precedence. |
| test/features/universal_import/data/parsers/macdive_sqlite_real_sample_test.dart | Adds gated real-sample regression test for MacDive SQLite parsing. |
| test/features/universal_import/data/parsers/macdive_sqlite_parser_test.dart | Parser unit tests for valid/invalid SQLite inputs. |
| test/features/universal_import/data/models/import_enums_test.dart | Updates enum/value counts and display names for new format/override option. |
| test/features/import_wizard/data/adapters/universal_adapter_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/import_wizard/data/adapters/healthkit_adapter_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/import_wizard/data/adapters/dive_computer_adapter_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/import_wizard/data/adapters/dive_computer_adapter_reimport_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/dive_import/presentation/providers/dive_import_notifier_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/dive_import/data/services/uddf_entity_importer_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/features/dive_computer/data/services/dive_import_service_test.mocks.dart | Updates mocks for new getSourceUuidByDiveId. |
| test/core/utils/bplist/bplist_decoder_test.dart | Golden + validation tests for the new bplist decoder. |
| lib/features/universal_import/presentation/providers/universal_import_providers.dart | Wires MacDive SQLite detection + parser selection + passes UUID map to dup checker. |
| lib/features/universal_import/data/services/macdive_raw_types.dart | Introduces typed raw row models + logbook container for MacDive SQLite reads. |
| lib/features/universal_import/data/services/macdive_dive_mapper.dart | Maps MacDive row graph into the unified ImportPayload shape. |
| lib/features/universal_import/data/services/macdive_db_reader.dart | Reads MacDive SQLite into typed row graph; schema validation + junction aggregation. |
| lib/features/universal_import/data/services/import_duplicate_checker.dart | Adds “pass 0” exact match by sourceUuid before fuzzy matching. |
| lib/features/universal_import/data/parsers/macdive_sqlite_parser.dart | New ImportParser orchestrating schema check + reader + mapper + error payloads. |
| lib/features/universal_import/data/models/import_enums.dart | Adds ImportFormat.macdiveSqlite and source override option. |
| lib/features/import_wizard/data/adapters/universal_adapter.dart | Passes UUID map to duplicate checker during wizard flow. |
| lib/features/dive_log/data/repositories/dive_repository_impl.dart | Adds getSourceUuidByDiveId() bulk helper for import dedup. |
| lib/core/utils/bplist/bplist_object.dart | Adds bplist AST types + convenience accessors. |
| lib/core/utils/bplist/bplist_decoder.dart | Implements bplist00 decoder (subset) used for MacDive Core Data blobs. |
| docs/superpowers/plans/2026-04-21-macdive-sqlite-import.md | Updates implementation plan/status notes and the ZSAMPLES descope. |
| CHANGELOG.md | Documents new MacDive SQLite import and limitations. |
ericgriffin
added a commit
that referenced
this pull request
Apr 23, 2026
…, UUID scoping) - bplist: UID now reads 1-4 bytes big-endian per spec (low nibble = byteCount - 1); regression tests added for 1/2/4-byte indexes and the BPlistUID docstring now matches the decoder. - MacDive DB reader: wrap optional-table reads (_readBuddies / _readTags / _readGear / _readTanks) in _selectOrEmpty so missing tables yield empty collections, matching the docstring; docstring tightened to call out the required/optional split explicitly. - dive_repository.getSourceUuidByDiveId: optional diverId parameter that joins to dives.diver_id, matching the scope the universal_adapter already uses for existingDives. 4 targeted tests. - CHANGELOG: rewrote the MacDive SQLite entry to describe only what actually ships (dives/sites/buddies/tags/gear plus per-dive tank/gas linkage); added a Known Limitations bullet for the critters/events/service-records/certifications gap. Inline code comments in the mapper document why dive dateTime follows M2's absolute-UTC convention and why diveToGearPks is collected-but-not- emitted, both tracked as cross-parser follow-up work. - real-sample test: replaced hard-coded dev path with MACDIVE_SQLITE_REAL_SAMPLE_PATH env var; group-level skip with a clear reason when unset; StateError guard for --run-skipped. - synthetic fixture: ZTANKANDGAS UUIDs renamed tag-uuid-* to tankandgas-uuid-*. - Regenerated 9 mock files for the new diverId parameter.
ericgriffin
added a commit
that referenced
this pull request
Apr 23, 2026
…robe dedup, doc fixes) Round 2 of Copilot's review on PR #256 — 7 items, 5 implemented and 2 doc/rename fixes. - equipmentRefs (the big one): my earlier pushback was wrong. `UddfEntityImporter` DOES resolve `diveData['equipmentRefs']` via `equipmentIdMapping`, so SQLite imports now emit per-dive equipmentRefs pulled from `diveToGearPks`. Each gear item gets a stable `uddfId` (MacDive UUID, with name fallback) so the importer can link gear back to dives the same way M2 XML would. Also filters name-less gear rows so `_importEquipment` doesn't silently drop entries while `equipmentIdMapping` keeps stale refs. - Format detector: probe SQLite table names ONCE per input, not once per flavor check. Previously detection wrote the full byte array to a temp file + opened sqlite twice (once for Shearwater, once for MacDive). Added `probeSqliteTableNames` on ShearwaterDbReader plus sync `matchesTables` on both readers so the detector can probe once and match all flavors. - Doc fix: `BPlistDecoder.decode` docstring no longer lists UIDs as unsupported — they've been supported since M3. - Doc fix: explicit comment at the bplist int-decode case explaining the 16-byte best-effort low-64-bit truncation behavior (implicit via Dart int overflow on `(value << 8) | byte`). - Doc+code: `MacDiveDbReader.readAll` now validates required tables up front and throws a clear FormatException if any are missing, so the docstring's "validated up-front" promise matches reality even for callers that bypass `isMacDiveDb`. - Rename: `samplesBplist` and `rawDataBplist` → `samplesBlob` / `rawDataBlob`. The data in these fields is NOT bplist (ZSAMPLES is MacDive-proprietary, ZRAWDATA is raw sensor dump); the old names would have misled future callers into trying to decode them. - Tests: 2 new synthetic-DB parser tests covering uddfId emission and equipmentRefs resolution; 1 new real-sample test verifying every emitted equipmentRef points at an equipment uddfId.
ericgriffin
force-pushed
the
feature/macdive-native-xml
branch
from
af121c3 to
28227e3
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Adds direct import from MacDive's Core Data SQLite database, the most
complete MacDive metadata path. Captures the same entity set as MacDive
XML — dives, sites, buddies, tags, and gear inventory — plus per-dive
tank/gas mix linkage from ZTANKANDGAS and per-dive equipmentRefs
linking dives to their gear. Cross-format deduplication via source UUIDs
means re-importing the same dives via MacDive UDDF/XML/SQLite won't
create duplicates.
## What landed
- `ImportFormat.macdiveSqlite` with `MacDive (SQLite)` source override.
- `_detectFormat` extended: SQLite → Shearwater check → MacDive check →
generic. Single SQLite probe reused across flavors to avoid doubling
temp-file I/O on large DBs.
- `BPlistDecoder` (Apple binary plist v00) in `lib/core/utils/bplist/`
— supports dict, array, string (ASCII + UTF-16BE), int (1..8 byte;
16-byte best-effort low-64-bit), real, bool, null, data, date, and
NSKeyedArchiver UID markers (1..4 bytes per spec).
- `MacDiveDbReader`: schema validation (required tables enforced up
front; optional tables read safely), typed row graph, filters null-FK
tombstones in ZTANKANDGAS.
- `MacDiveDiveMapper`: joins raw rows into a unified `ImportPayload`
matching M2's key conventions. Emits per-dive `equipmentRefs` so
`UddfEntityImporter.equipmentIdMapping` can link gear back to dives.
Gear maps require `name` and carry a stable `uddfId` (MacDive gear
UUID, with name fallback).
- `MacDiveSqliteParser`: `ImportParser` implementation wrapping reader
+ mapper.
- `ImportDuplicateChecker`: first-pass exact match on `source_uuid`
via a new `DiveRepository.getSourceUuidByDiveId({diverId})` helper
that optionally scopes the UUID map to one diver.
## Known limitations
- `ZDIVE.ZSAMPLES` (MacDive's proprietary profile-sample BLOB) is NOT
decoded. MacDive's format isn't bplist and doesn't match common
compressions. Users who want profile time-series data should use
MacDive UDDF import (M1). M3 is the rich-metadata path; UDDF is the
sample-data path.
- Critters (marine-life sightings), dive events, service records, and
certifications are read into the typed row graph but not yet emitted
into the import payload — tracked as follow-up work (the unified
importer lacks entity types for critters/events/service records).
## Test coverage
- Synthetic-DB tests: reader, mapper, parser (`macdive_db_reader_test`,
`macdive_dive_mapper_test`, `macdive_sqlite_parser_test`).
- BPlist decoder golden tests (Python-plistlib fixtures + real MacDive
ZTIMEZONE BLOB). Regression tests for 1/2/4-byte UID indexes.
- Gated real-sample regression against a 6.7MB MacDive.sqlite (enable
via `MACDIVE_SQLITE_REAL_SAMPLE_PATH`): 540 dives / 373 sites /
33 buddies / 39 tags / 32 gear, equipmentRefs resolution verified.
- New `dive_repository_new_methods_test` coverage for scoped and
unscoped `getSourceUuidByDiveId`.
Closes #179.
ericgriffin
force-pushed
the
feature/macdive-sqlite
branch
from
9512cfb to
eea760b
Compare
github-project-automation
Bot
moved this from In review
to Done
in Submersion Release Tracker
6 tasks
ericgriffin
added a commit
that referenced
this pull request
Apr 23, 2026
Follow-up to the MacDive SQLite importer (PR #256) which emits profile: [] for every dive. Spec proposes a two-phase approach: investigation spike with a go/no-go gate, then decoder implementation if the spike confirms feasibility. Stacks on feature/macdive-sqlite.
ericgriffin
added a commit
that referenced
this pull request
Apr 23, 2026
6 tasks
readme42
pushed a commit
to readme42/submersion
that referenced
this pull request
Apr 24, 2026
MacDiveDiveMapper now decodes ZRAWDATA via parseRawDiveData (same API ShearwaterDiveMapper uses for Shearwater Cloud). toPayload becomes async and threads warnings through. _vendorProductFromZComputer maps ZCOMPUTER strings to libdivecomputer (vendor, product) pairs. Sample projection matches ShearwaterDiveMapper.mergeWithParsedDive exactly (same keys, same conditional emission). Dives without ZRAWDATA or with unmapped computers emit profile:[] silently; decode failures emit profile:[] + ImportWarning. Closes the ZRAWDATA part of the MacDive SQLite profile gap from submersion-app#256.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third of four milestones addressing MacDive import issues. Closes
GH #179 (MacDive SQLite import). Gives users the most complete
migration path from MacDive: all dive metadata plus tags, critters,
gear, events, service records, certifications.
Stacked on PR #254 (M2). Merge order: #252 → #254 → this.
What landed
ImportFormat.macdiveSqlitewithMacDive (SQLite)source override._detectFormatextended: SQLite → Shearwater check → MacDive check → generic.BPlistDecoder(Apple binary plist v00) inlib/core/utils/bplist/— supports dict, array, string (ASCII + UTF-16BE), int, real, bool,
null, data, date, and (added during real-sample probing) UID markers
for NSKeyedArchiver format.
MacDiveDbReader: schema validation, typed row graph, filters null-FKtombstones in ZTANKANDGAS.
MacDiveDiveMapper: joins raw rows into a unifiedImportPayloadmatching M2's key conventions. Reuses
MacDiveUnitConverterandMacDiveValueMapperfrom M2.MacDiveSqliteParser:ImportParserimplementation wrapping readerImportDuplicateChecker: first-pass exact match onsource_uuidvia a new bulk-query helper (
DiveRepository.getSourceUuidByDiveId),keeps
Diveentity unchanged.32 gear against user's 6.7MB DB.
Known limitation
ZDIVE.ZSAMPLES(MacDive's proprietary profile-sample BLOB) is NOTdecoded in M3. MacDive's format isn't bplist and doesn't match any
common compression at any offset (tried zlib/gzip/lzma at 0/4/8/12).
Users who want profile time-series data should use MacDive UDDF
import (M1) — that decodes MacDive's UDDF profile correctly.
M3 is the "rich metadata" path; UDDF is the "sample data" path.
Test plan
flutter test— full suite passesflutter analyze— cleandart format— cleanflutter test --tags=real-data --run-skippedagainst user's 6.7MB MacDive.sqliteRelated
source_uuidcolumn +IncomingDiveDatafields).MacDiveValueMapper+MacDiveUnitConverter).What's still deferred
and M3's SQLite reader to emit
imageRefson payloads, adds aphoto-linking wizard step with path rebasing. Separate plan at
docs/superpowers/plans/2026-04-21-macdive-photo-import.md.