Extend metadata concept resolution and RAW handling

Add lens-correction and RAW-processing concept kinds/roles and preserve grouped matrix/vector/table values in concept candidates. Introduce has_values/values in MetadataConceptCandidate, copy/merge/value-key helpers, and numeric/date tolerance-aware conflict checks (GPS coordinate/altitude tolerances). Treat candidates that share source entries as non-conflicting. Expand vendor RAW/source-processing classification (pixel-shift, multi-shot, composite, auto-lighting) and update query/interpretation plumbing to emit full normalized value vectors. Update docs and tests to reflect improved coverage and behavior.

Author: ssh4net

CHANGES.md

@@ -113,13 +113,25 @@ Changes compared with `0.4.8`.
   active area, border, and sensor-geometry records. Geometry candidates expose
   canonical origin, size, rect, and margin fields when the query/interpretation
   layer can normalize them.
+- Extended cross-family concept resolution with full normalized value vectors
+  and new lens-correction and RAW-processing concept families. Color/white
+  balance, lens-correction, black/white level, linearization, CFA layout,
+  raw-storage, and source-processing concepts now preserve grouped
+  interpretation values instead of only the first scalar preview values.
+- GPS concept conflict checks now compare numeric coordinates and altitude with
+  explicit tolerances, while grouped candidates that share source entries are
+  treated as alternate views of the same evidence rather than conflicts.
 - Semantic crop queries now expose canonical border-margin candidates for
   parseable border/padding XMP text, DNG masked-area candidates, and Phase
   One/Leaf geometry margins.
 - Vendor RAW/source-processing classification now distinguishes source-private
   preview, face-geometry, computational, thermal, and stitch/panorama buckets
   in addition to the existing color, white-balance, geometry, storage,
   lens-correction, raw-data, sensor, and private-table groups.
+- Vendor RAW/source-processing classification now also treats common
+  computational MakerNote terms such as pixel-shift, multi-shot, composite, and
+  auto-lighting optimizer fields as source-private processing metadata for
+  audit and rendered-transfer safety decisions.
 - Added focused regression coverage for compatible-file versus rendered-image
   transfer safety: compatible mode keeps serializable source RAW/camera-specific
   metadata, while rendered mode drops source-specific metadata and uses

docs/api_stability.md

@@ -26,8 +26,8 @@ different status.
 | EXIF/TIFF/DNG numeric value names: `exif_tag_numeric_value_name(...)` and focused helpers | `openmeta/exif_value_names.h` | Stable | Small helper contract for common enum-like TIFF/EXIF/DNG numeric values such as compression, photometric interpretation, planar configuration, exposure program, metering mode, light source, flash, color space, white balance, scene capture type, gain control, CFA layout, and DNG calibration illuminants. Unknown values return an empty string and remain lossless numeric metadata. |
 | Semantic metadata query: `query_metadata(...)`, `query_crop_metadata(...)`, focused query helpers, and `metadata_query_fuzzy_search_available()` | `openmeta/metadata_query.h` | Experimental | Query contract for inspection matches plus normalized candidates. Current coverage includes crop/active-area/border margins, exposure/gain, white balance, color, lens correction, orientation, and RAW/source-processing metadata across standard tags, selected DNG tags, fuzzy XMP paths, and vendor RAW-processing classification. Matches report `exact_match`, `fuzzy_match`, and `fuzzy_score` so tools can label exact results separately from RapidFuzz near-miss hits. `OPENMETA_ENABLE_RAPIDFUZZ=ON` adds optional near-miss XMP/property-path scoring. Grouped candidates include `matrix_set`, `vector_set`, and `table` shapes for related non-crop metadata, including RAW black/white levels, linearization, CFA/sensor layout, source geometry, raw-storage identifiers, and source-processing buckets. Python `Document` and `TransferSourceSnapshot` mirror this as thin dictionary-returning wrappers. |
 | Structured metadata interpretation records: `interpret_metadata(...)`, `interpret_metadata_query(...)` | `openmeta/metadata_interpretation.h` | Experimental | Thin structured projection over semantic query candidates. Records carry query class, semantic kind, normalized shape, confidence, source entry ids, and normalized origin/size/rect/margins/value arrays where available. Current scope covers orientation, geometry/crop/border, exposure/gain, color/white-balance, lens-correction, and RAW/source-processing records. Python `Document` and `TransferSourceSnapshot` expose matching dictionary wrappers. |
-| Cross-family concept resolution: `resolve_metadata_concepts(...)`, `resolve_metadata_concept(...)` | `openmeta/metadata_concepts.h` | Experimental | First bounded resolver for duplicated host-facing concepts. Current scope reports candidates, candidate source entries, source families, preferred entries, normalized numeric/text keys, normalized date/time fields, date/time precision, timezone kind, normalized geometry fields, and same-role conflicts for orientation, date/time, color/profile, GPS, and geometry evidence across EXIF, XMP, IPTC, ICC, PNG text, and query-backed interpretation records where applicable. Geometry candidates cover crop, active area, border, and sensor geometry with canonical origin, size, rect, and margin fields when available. GPS date/time is combined from `GPSDateStamp` plus `GPSTimeStamp` when both entries exist, and GPS altitude candidates expose altitude-reference code plus below-sea-level state when reference metadata is present. It is intended for inspection UI and host policy decisions; it does not rewrite metadata or hide ambiguity. Python `Document` and `TransferSourceSnapshot` expose matching dictionary wrappers. |
-| Vendor RAW-processing summaries: `vendor_raw_processing_from_store(...)`, `classify_vendor_raw_processing_field(...)` | `openmeta/vendor_raw_processing.h` | Experimental | Conservative grouped source-RAW/source-processing field summaries for decoded Sony, Canon, Nikon, Fujifilm, Pentax, Panasonic, Olympus, Kodak, Minolta, Sigma, Samsung, Ricoh, Apple, DJI, Google, FLIR, Casio, Sanyo, KyoceraRaw, Reconyx, HP, JVC, GE, Motorola, Nintendo, and Microsoft MakerNotes, including vendor-private, computational, thermal, preview, face-geometry, stitch/panorama, Apple computational capture/HDR/motion, DJI pose/thermal, Google HDR+/shot-log, and FLIR radiometric/raw-value buckets. Intended for audit/UI and rendered-transfer safety decisions, not for writing vendor RAW/source-processing values into rendered targets. |
+| Cross-family concept resolution: `resolve_metadata_concepts(...)`, `resolve_metadata_concept(...)` | `openmeta/metadata_concepts.h` | Experimental | First bounded resolver for duplicated host-facing concepts. Current scope reports candidates, candidate source entries, source families, preferred entries, normalized numeric/text keys, full normalized value vectors, normalized date/time fields, date/time precision, timezone kind, normalized geometry fields, and same-role conflicts for orientation, date/time, color/profile, GPS, geometry, lens-correction, and RAW-processing evidence across EXIF, XMP, IPTC, ICC, PNG text, and query-backed interpretation records where applicable. Geometry candidates cover crop, active area, border, and sensor geometry with canonical origin, size, rect, and margin fields when available. Color/white-balance, lens-correction, and RAW-processing concepts preserve grouped matrix/vector/table values for host inspection; they do not make source-bound values safe to serialize into rendered targets. GPS date/time is combined from `GPSDateStamp` plus `GPSTimeStamp` when both entries exist, and GPS altitude candidates expose altitude-reference code plus below-sea-level state when reference metadata is present. It is intended for inspection UI and host policy decisions; it does not rewrite metadata or hide ambiguity. Python `Document` and `TransferSourceSnapshot` expose matching dictionary wrappers. |
+| Vendor RAW-processing summaries: `vendor_raw_processing_from_store(...)`, `classify_vendor_raw_processing_field(...)` | `openmeta/vendor_raw_processing.h` | Experimental | Conservative grouped source-RAW/source-processing field summaries for decoded Sony, Canon, Nikon, Fujifilm, Pentax, Panasonic, Olympus, Kodak, Minolta, Sigma, Samsung, Ricoh, Apple, DJI, Google, FLIR, Casio, Sanyo, KyoceraRaw, Reconyx, HP, JVC, GE, Motorola, Nintendo, and Microsoft MakerNotes, including vendor-private, computational, thermal, preview, face-geometry, stitch/panorama, Apple computational capture/HDR/motion, DJI pose/thermal, Google HDR+/shot-log, pixel-shift/multi-shot/composite/auto-lighting source processing, and FLIR radiometric/raw-value buckets. Intended for audit/UI and rendered-transfer safety decisions, not for writing vendor RAW/source-processing values into rendered targets. |
 | Transfer safety audit: `transfer_safety_audit_from_store(...)` | `openmeta/metadata_transfer.h` | Experimental | Preflight summary of source entries and entries filtered or invalidated by `TransferSafetyMode`, including Sony/Canon/Nikon/Fujifilm/Pentax/Panasonic/Olympus/Kodak/Minolta/Sigma/Samsung/Ricoh/Apple/DJI/Google/FLIR/Casio/Sanyo/KyoceraRaw/Reconyx/HP/JVC/GE/Motorola/Nintendo/Microsoft RAW/source-processing buckets. Intended for diagnostics and host UI before preparing rendered-image transfers. |
 | Raw-carrier passthrough audit: `raw_carrier_passthrough_audit_from_snapshot(...)` | `openmeta/metadata_transfer.h` | Experimental | Diagnostic preflight for opt-in raw carriers. Reports candidate carriers and primary block reasons such as missing payload, target incompatibility, safety filtering, content-bound C2PA, explicit profile policy, missing decoded-entry links, or unsupported carrier kind. Hosts can call it directly before enabling snapshot passthrough. |
 | Source snapshot type and read helpers: `TransferSourceSnapshot`, `read_transfer_source_snapshot_file(...)`, `read_transfer_source_snapshot_bytes(...)`, `build_transfer_source_snapshot(...)` | `openmeta/metadata_transfer.h` | Experimental | Current snapshots are decoded-store-backed by default. Opt-in raw carriers preserve bounded source payload/provenance records and snapshot-local decoded entry ids for host diagnostics and bounded passthrough decisions. Const reuse is safe when callers do not mutate the snapshot and do not share returned result objects across writers. |

docs/development.md

@@ -17,8 +17,8 @@ model should stay compact:
 | Area | Purpose | Readiness |
 | --- | --- | --- |
 | Decoding | Find metadata carriers and decode EXIF, XMP, IPTC, ICC, Photoshop IRB, JUMBF/C2PA, EXR, and related blocks into `MetaStore` entries. | High, about 98-100% for the current target scope. |
-| Interpretation | Normalize names and values, group entries by meaning, and classify source-bound data such as RAW crop, color, lens-correction, sensor, and vendor-private fields. | Medium-high, about 80%. |
-| Query | Find entries by name, fuzzy term, or semantic group, then expose normalized query candidates, structured interpretation records, and bounded cross-family concept resolutions for crop/border/active-area, exposure/gain, color/WB, orientation, date/time, GPS, lens-correction, and RAW/source-processing fields across standard and vendor metadata. | Medium, about 50-60%. |
+| Interpretation | Normalize names and values, group entries by meaning, and classify source-bound data such as RAW crop, color, lens-correction, sensor, and vendor-private fields. | Medium-high, about 82%. |
+| Query | Find entries by name, fuzzy term, or semantic group, then expose normalized query candidates, structured interpretation records, and bounded cross-family concept resolutions for crop/border/active-area, exposure/gain, color/WB, orientation, date/time, GPS, lens-correction, and RAW/source-processing fields across standard and vendor metadata. | Medium, about 63-70%. |
 | Creation | Build fresh metadata entries from host-provided values. | Medium, about 55-65%. |
 | Editing | Modify existing logical metadata entries while preserving valid surrounding structure. | Medium, about 60-70%. |
 | Transfer | Move metadata between files using explicit compatible-file or rendered-image safety policies. | Medium-high, about 80-85%. |
@@ -64,13 +64,15 @@ into records with query class, semantic kind, normalized shape, confidence,
 source entries, and normalized geometry/value arrays where available.
 
 For cross-family duplicated concepts, use `openmeta/metadata_concepts.h`.
-It currently resolves orientation, date/time, color/profile, GPS, and geometry
-into candidate lists with candidate source entries, source families, preferred
-entries, normalized compare keys, parsed date/time fields, date/time precision,
-timezone kind, GPS altitude-reference state, canonical geometry
-origin/size/rect/margins, and same-role conflict flags. This is deliberately an
-inspection/policy surface; host code still decides whether a conflict should be
-shown, ignored, or corrected during editing/transfer.
+It currently resolves orientation, date/time, color/profile, GPS, geometry,
+lens-correction, and RAW-processing into candidate lists with candidate source
+entries, source families, preferred entries, normalized compare keys, parsed
+date/time fields, date/time precision, timezone kind, GPS altitude-reference
+state, canonical geometry origin/size/rect/margins, full normalized value
+vectors for grouped matrix/vector/table records, and same-role conflict flags.
+This is deliberately an inspection/policy surface; host code still decides
+whether a conflict should be shown, ignored, or corrected during
+editing/transfer.
 
 ## Build Prerequisites
 

docs/host_integration.md

@@ -50,15 +50,19 @@ confidence, and normalized geometry/value arrays.
 
 For host code that needs to reconcile duplicated concepts across metadata
 families, use `metadata_concepts.h`. It reports orientation, date/time,
-color/profile, GPS, and geometry candidates with source families, preferred
-entries, and same-role conflict flags. Geometry candidates expose crop,
-active-area, border, and sensor-geometry roles with canonical origin, size,
-rect, and margin fields when available. Date/time candidates include parsed
-date/time fields when the source value is recognizable, plus precision and
-timezone-kind fields. GPS timestamps combine `GPSDateStamp` with
+color/profile, GPS, geometry, lens-correction, and RAW-processing candidates
+with source families, preferred entries, and same-role conflict flags. Geometry
+candidates expose crop, active-area, border, and sensor-geometry roles with
+canonical origin, size, rect, and margin fields when available. Color/white
+balance, lens-correction, and RAW-processing candidates expose full normalized
+value vectors for grouped matrix/vector/table records. Date/time candidates
+include parsed date/time fields when the source value is recognizable, plus
+precision and timezone-kind fields. GPS timestamps combine `GPSDateStamp` with
 `GPSTimeStamp` when both are present, and GPS altitude candidates report
 whether `GPSAltitudeRef` marked the height as below sea level. Treat this as an
-inspection and policy input rather than an automatic metadata rewrite decision.
+inspection and policy input rather than an automatic metadata rewrite decision;
+source-bound color, lens, and RAW-processing values still need rendered-transfer
+safety filtering.
 
 ## Adapter Classes
 
@@ -83,7 +87,7 @@ OpenMeta splits host integration surfaces deliberately:
   `metadata_interpretation.h` for query-backed semantic records
 - concept-resolution utility:
   `metadata_concepts.h` for cross-family orientation, date/time, color/profile,
-  GPS, and geometry conflict inspection
+  GPS, geometry, lens-correction, and RAW-processing conflict inspection
 
 ## 1. Read Into `MetaStore`