Timeline resource streaming

[AaronPlave]

Summary

This PR implements streaming of simulation resources and external datasets (profiles) from the database as they are available, closes #1650 . External datasets are now fetched on an as-needed basis instead of fetching all external datasets that could possibly be associated with the plan every time planDataset subscription changed, closes #352.

• Sim profiles stream segments via a new createProfileSubscription factory keyed on (datasetId, name). Each simulationDataset update triggers GET_PROFILE_SINCE for only the segments past the last seen start_offset.

• External dataset profiles stream the same way via a new createExternalResourceSubscription factory. Metadata source is planDatasets (live subscription, already in use). When a profile's duration advances, the factory pulls only the new segments via GET_EXTERNAL_PROFILE_SEGMENTS_SINCE.

• Removes the batch external profiles fetch. GET_PROFILES_EXTERNAL, effects.getResourcesExternal, the externalResources/fetchingResourcesExternal writable stores, and the +page.svelte reactive statement that re-fetched all external profiles on every planDatasets push are all gone.

• Fixes external profile start time anchor modify external profile start time to use plan start time instead of sim start time so that external profiles draw correctly in time when plan is using subset sim.

• New TimelineStatusIndicator next to the Timeline header. Aggregates loading/error across profile subscriptions, span fetches, constraint runs, external-event raw subscriptions, and active simulation status. Tooltip shows what failed (sim "Profile X" vs external "External profile X").

• stores/timelineResourceStatus.ts loading/error registry for any resource on the timeline.

• allResourceTypes now derives from planDatasets metadata directly (filtered to plan-level + current-sim rows) instead of the now removed batch fetch.

• External event row loading states loading/error states on rows containing external events. Note that this is derived from the global external sources store since there are no individual fetches or subscriptions for singular derivation groups so if one is loading/failed then all will share that state.

Tests

• New: stores/profile.test.ts, stores/externalResource.test.ts, utilities/profile.test.ts. Cover settling→found, race-defer, settled-missing, sim-tied vs plan-level row preference, duration-advance refetch, profile-row switch reset, refetch error surface, dispose-during-in-flight.
• Expanded: utilities/resources.test.ts with sort-invariant regression pins on sampleProfiles.
• New e2e: simulation.test.ts Streaming pipeline smoke test — asserts the indicator settles and timeline canvases render non-transparent content across two re-sims. Shared canvas-pixel helper in e2e-tests/utilities/canvas.ts is now used by plan-external-source.test.ts too.

Test plan

• Configure backend to run orbiter model by mounting spice kernels as specified here https://github.com/NASA-AMMOS/plandev-examples/tree/main/examples/05-orbiter
• Load a plan that produces thousands of profile segments such as the demo plan from the Orbiter model. Ensure that resources load in progressively during simulation and reach the end of the simulation once complete. Refresh the page and ensure that the simulation results look identical to the end-of-streaming state.
• Load a basic model that simulates quickly like Banananation and verify that row data still loads quickly before and during simulation.
• Upload a multi-thousand-segment external dataset: row plot fills in progressively, timeline indicator stays visible while sim is active.
• Open a plan whose view references a resource that doesn't exist in the current model: row shows "Failed to load profiles for 1 layer", indicator shows error icon with "Resource not found in attached external datasets" tooltip.
• Run a failed sim (e.g. invalid activity): indicator settles once status is Failed.

[duranb]

works great!

[dandelany]

I've been testing this with @AaronPlave and we noticed something - the UI streaming works fine for normal simulation runs, but not for scheduling runs which run simulation at the end. For a (relatively) big plan, we're able to click simulate and see all the profiles slowly load in for the normal case. When I click "Schedule" with some goals in place, I see a couple of profiles load completely first, the sim hangs at "0%" for awhile, then the full remaining profiles all load at once.

@Mythicaeda does this ring any bells for you? Do simulations triggered by scheduling stream their profiles to the backend the same way that normal runs do? Or does this seem like a likely UI bug to you?

[dandelany]

Testing note - the timeline loading indicator at the top of the timeline is shown when simulation is running, but not during Scheduling. Per discussion we think it's best to also show it during Scheduling, since that is what is ultimately blocking the Timeline from showing up.

@AaronPlave will get to this tomorrow if he has time - otherwise we consider this a nonblocker for later

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[AaronPlave]

Testing note - the timeline loading indicator at the top of the timeline is shown when simulation is running, but not during Scheduling. Per discussion we think it's best to also show it during Scheduling, since that is what is ultimately blocking the Timeline from showing up.

@AaronPlave will get to this tomorrow if he has time - otherwise we consider this a nonblocker for later

Done

[Mythicaeda]

@Mythicaeda does this ring any bells for you? Do simulations triggered by scheduling stream their profiles to the backend the same way that normal runs do? Or does this seem like a likely UI bug to you?

@dandelany This is not a UI bug. Scheduling does not stream to the database.

Explanation as to why:
Currently, the Scheduler does not distinguish between a "working simulation" (the simulations that are occurring either between or in the middle of goals) and a "post-scheduling simulation". To it, a "post-scheduling simulation" is just "the last requested working simulation". As a result, if the scheduler needs to resimulate the plan to generate the "post-scheduling simulation results" (it might not need to), it'll use the same InMemoryResourceManager that it uses in working simulations, rather than trying to swap to the StreamingResourceManager.

(Hopefully it's intuitively understandable why we don't stream the scheduler's "working simulations" to the database, but just in case I'll give a couple reasons:

1. Most importantly, the scheduler needs to reference the resource profiles of these working sims. Since the point of streaming is to free up memory, our StreamingResourceManager only holds onto these profiles long enough to build up a batch.
2. It avoids cluttering the DB, especially since working sims can reference activities that aren't actually going to make it into the database (ie, if a goal is calling sim to test an activity placement, but determines that that placement is wrong and rolls back the change).
3. Working sims don't follow necessarily go from start of sim horizon to end of sim horizon. In fact, the scheduler is designed to find an applicable CheckpointSimulationEngine that is partway into the sim horizon and pick up simulation from there.)