feat(1012): Tag ingestion pipeline — raw files → per-tag .mat (batch + live)#59
Merged
Conversation
…-06, D-11) Merges plan 1012-02 source from worktree agent-a550e129.
…(D-01/D-02/D-04/D-06/D-09/D-10/D-11) readRawDelimited_, selectTimeAndValue_, writeTagMat_ under libs/SensorThreshold/private/; readRawDelimitedForTest_ public test shim (Major-1 Option A) so tests/suite/ can exercise private helpers past MATLAB private-folder scoping. All 18 TestRawDelimitedParser placeholders now GREEN.
Mid-task checkpoint (Minor-2 / revision-1) — class skeleton that enumerates ingestable tags but does not yet ingest. - classdef BatchTagPipeline < handle with public OutputDir/Verbose properties + SetAccess=private LastReport/LastFileParseCount (Major-2 observability property declared, initialised to 0, wiring deferred to the run() commit) - constructor with inline NV-parse (no parseOpts dep — private/ across libs is unreachable), auto-mkdir on missing OutputDir, throws TagPipeline:invalidOutputDir / TagPipeline:cannotCreateOutputDir - isIngestable_ static private predicate: POSITIVE isa-check on SensorTag/StateTag only (D-16 / Pitfall 10 — MonitorTag/CompositeTag never materialised; Tag.m untouched) - eligibleTags_ routes TagRegistry.find to the predicate Next commit: run() loop + ingestTag_/parseOrCache_/dispatchParse_ + per-tag try/catch + end-of-run throw + test GREEN bodies.
…ne suite Second half of the Minor-2 / revision-1 two-commit checkpoint — completes Plan 04 by adding the ingestion loop and turning 18 RED placeholders GREEN. BatchTagPipeline.m additions (~99 new lines on top of the skeleton): - run(): per-run containers.Map fileCache_, try/catch per tag, end-of-run LastFileParseCount capture BEFORE cache reset (Major-2 observability), and TagPipeline:ingestFailed throw when any tag failed (D-18) - ingestTag_: rs -> abspath -> parseOrCache_ -> selectTimeAndValue_ - parseOrCache_: containers.Map isKey -> cached, else dispatchParse_ then cache (D-07 dedup hotspot; LastFileParseCount reads .Count here) - dispatchParse_: D-02 hidden extension switch .csv/.txt/.dat -> readRawDelimited_, else TagPipeline:unknownExtension - absPath_: pwd-relative fallback so fileCache_ keys are stable across tag-order permutations Test suite (18 GREEN tests -- full decision matrix): - D-15 / D-19: testConstructorRequiresOutputDir, testConstructorCreatesOutputDirIfMissing, testErrorCannotCreateOutputDir - D-04: testWideFileFanOut, testTallFileTwoColumn - D-09: testRoundTripThroughSensorTagLoad (SensorTag.load recovers X/Y) - D-10: testOneMatFilePerTag (3 distinct <Key>.mat files) - D-11: testStateTagCellstrRoundTrip (cellstr Y preserved) - D-07 + Major-2: testFileCacheDedup asserts LastFileParseCount == 1 after 2 tags share a single RawSource.file - D-08 + D-16: testSilentSkipMonitorTag, testSilentSkipTagWithoutRawSource, testCompositeTagNotMaterialized - D-17: testMonitorPersistPathUntouched (MonitorTag.recomputeCount_ stays at 0 through run() -- pipeline never routes a MonitorTag through the parser+writer, Persist path untouched) - D-18: testPerTagErrorIsolationContinuesToNext, testIngestFailedThrownAtEnd - D-19: testErrorInvalidRawSource (re-asserts Plan 02 validator), testErrorInvalidWriteMode (re-asserts Plan 03 writer), testDispatchUnknownExtension (unknown-ext via .xml trips TagPipeline:unknownExtension through the ingestion try/catch) Grep-gate verification (all passing): - readRawDelimitedForTest_ in BatchTagPipeline.m: 0 (production isolation) - negative isa on MonitorTag/CompositeTag in BatchTagPipeline.m: 0 (D-16 / Pitfall 10 -- positive-isa predicate only) - positive isa on SensorTag/StateTag: 1 (isIngestable_ branch) - readtable/readmatrix/readcell/detectImportOptions in libs/SensorThreshold/: 0 (Octave parity preserved -- textscan only via readRawDelimited_) - '-append' in libs/SensorThreshold/: 0 (Pitfall 2 -- writeTagMat_ uses load -> concat -> save, never save -append) - TagRegistry.find usage: 1 (enumeration gateway) - containers.Map usage: 3 (fileCache_ init + reset + isKey guard) - LastFileParseCount in class: 3 / in test: 3 File touches: 2 of 12 budget (BatchTagPipeline.m new, TestBatchTagPipeline.m edited). Cumulative phase count: 11 / 12 after this commit.
- 1012-04-SUMMARY.md: full deviations log (3 auto-fixed per Rules 1/2/3), error-ID coverage table, round-trip proof sketch, 14-gate grep audit, two-commit checkpoint record, self-check PASSED - STATE.md: advance plan counter to 2 of 5; progress 97%; record-metric for 1012-04 (12min, 1 task, 2 files); 4 decisions added to Accumulated Context; session resume file cleared - ROADMAP.md: Phase 1012 progress table updated (4/5 plans)
…EEN tests D-07 per-tick file-parse de-dup via tickCache + LastFileParseCount observability (Major-2 parity with BatchTagPipeline). D-12 shared helper path: reuses readRawDelimited_ / selectTimeAndValue_ / writeTagMat_ with the batch class. D-13 modTime+lastIndex state machine mirrors MatFileDataSource.fetchNew adapted from .mat arrays to text-file rows. D-14 classdef LiveTagPipeline < handle (NOT subclass of LiveEventPipeline; borrows only the timer ergonomics). D-15 OutputDir constructor param with auto-mkdir. D-16 positive-isa predicate on SensorTag/StateTag only (Pitfall 10 discipline). D-18 per-tag try/catch so one tag's failure does not abort the tick. D-19 error-ID taxonomy preserved. Research Q3: tagState_ entries GC'd each tick for tags no longer in TagRegistry; exposed via the Dependent TagStateCount property so testTagStateGCDropsUnregistered can observe it. Pitfall 2 gate: append mode delegates to writeTagMat_'s load->concat-> save path; no use of the dash-append flag anywhere in the class. Pitfall 4 gate: tests use pause(1.1) before re-touching raw files (mirrors TestMatFileDataSource). Pitfall 8 gate: stop() guards isvalid(timer_) before stop+delete inside try/catch so stop-during- tick cannot cascade. Octave parity fix: the eligibility predicate is expressed as an inline anonymous function inside eligibleTags_ rather than a handle to a static private method. Octave 7+ rejects cross-class private- method handles at call time from within TagRegistry.find, so the documented approach (handle to a private static) fails Octave parity. The inline lambda side-steps the reflection check entirely. The lambda body stays byte-semantically identical to the predicate used by BatchTagPipeline.isIngestable_; adding a new eligible tag kind requires updating both call sites. A pre-existing variant of this defect in BatchTagPipeline (plan 04) is logged to deferred-items.md for a follow-up plan. Test suite (11 tests, all GREEN on MATLAB; core tick semantics verified via Octave smoke-test since matlab.unittest has no Octave runner): - testNoSubclassOfLiveEventPipeline (D-14 via meta.class) - testConstructorRequiresOutputDir (D-19 TagPipeline:invalidOutputDir) - testStartSetsStatusRunning / testStopSetsStatusStopped (D-14 timer) - testFirstTickWritesAll (D-13 first tick reads all) - testSecondTickWritesOnlyNewRows (D-13 modTime+lastIndex incremental) - testUnchangedFileSkipped (D-13 modTime guard; LastFileParseCount=0) - testDedupAcrossTagsPerTick (D-07 + Major-2 LastFileParseCount==1) - testPerTagFileIsolation (D-10 under live writes) - testAppendModePreservesPriorRows (Pitfall 2 gate: [1 2 3]+[4 5]=[1..5]) - testTagStateGCDropsUnregistered (Research Q3 via TagStateCount) File-count ledger: 1 NEW (LiveTagPipeline.m, 357 lines) + edits to TestLiveTagPipeline.m (already counted in Plan 01) - phase total 12/12 at exact budget (Pitfall 5 margin=0).
- Add Plan 05 SUMMARY with 19-decision matrix, 3 deviations, grep-gate audit (per-class + phase-level), pitfall audit, and file-count ledger (12/12 exact). - Add deferred-items.md logging the pre-existing Plan 04 BatchTagPipeline Octave parity defect (cross-class @classname.staticPrivate handle rejection at TagRegistry.find call time) for a follow-up plan. - Update STATE.md: mark phase 1012 ready_for_verification (all 5 plans complete), append Plan 05 performance metric, record 4 decisions from Plan 05 execution, add deferred Octave-parity blocker. - Update ROADMAP.md: phase 1012 plan progress 5/5 Complete. Phase 1012 is feature-complete. All 19 decisions addressed, file budget 12/12 consumed exactly, Pitfall 5 margin = 0 as documented.
S-1 (Octave parity): BatchTagPipeline.eligibleTags_ now uses the same inline-lambda predicate as LiveTagPipeline, avoiding Octave's cross-class private-method-handle rejection. Dead `isIngestable_` private static method removed. deferred-items.md entry marked RESOLVED. S-2 (dead code): LiveTagPipeline.processTag_ had a byte-identical if/else on iscell(y) that did the same thing in both branches. Collapsed to a single `newY = y(newRange);`. S-3 (leading-blank-line edge case in readRawDelimited_) left as a follow-up comment target; unchanged.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'FastSense Performance'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.10.
| Benchmark suite | Current: fcf850c | Previous: 9be4738 | Ratio |
|---|---|---|---|
Downsample mean (1M) |
3.418 ms |
2.292 ms |
1.49 |
Downsample mean std(1M) |
0.06 ms |
0.051 ms |
1.18 |
Zoom cycle mean std(1M) |
4.587 ms |
3.975 ms |
1.15 |
Downsample mean (5M) |
16.579 ms |
11.004 ms |
1.51 |
Instantiation mean std(5M) |
3.725 ms |
0.303 ms |
12.29 |
Downsample mean (10M) |
34.703 ms |
21.704 ms |
1.60 |
Downsample mean std10M) |
2.617 ms |
0.232 ms |
11.28 |
Instantiation mean std10M) |
3.305 ms |
1.868 ms |
1.77 |
Zoom cycle mean std10M) |
1.31 ms |
0.68 ms |
1.93 |
Downsample mean (50M) |
163.842 ms |
108.651 ms |
1.51 |
Instantiation mean std50M) |
12.24 ms |
10.524 ms |
1.16 |
Render mean std50M) |
3.457 ms |
1.197 ms |
2.89 |
Downsample mean (100M) |
326.809 ms |
214.888 ms |
1.52 |
Downsample mean ( std00M) |
0.476 ms |
0.3 ms |
1.59 |
Zoom cycle mean ( std00M) |
1.399 ms |
1.115 ms |
1.25 |
Downsample mean (500M) |
1649.17 ms |
1097.99 ms |
1.50 |
Downsample mean ( std00M) |
35.016 ms |
0.3 ms |
116.72 |
Instantiation mean ( std00M) |
1202.268 ms |
38.558 ms |
31.18 |
Render mean ( std00M) |
548.843 ms |
7.961 ms |
68.94 |
Dashboard page switch mean |
0.178 ms |
0.126 ms |
1.41 |
Dashboard page switch stdmean |
0.161 ms |
0.061 ms |
2.64 |
Dashboard broadcastTimeRange stdmean |
0.029 ms |
0.024 ms |
1.21 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @HanSur94
…er teardown Pre-existing bug surfaced by PR #59's CI run (500k+ stderr loop in testTimerContinuesAfterError). stopLive() called stop(obj.LiveTimer); delete(obj.LiveTimer); BEFORE obj.IsLive=false — so any queued onLiveTimerError that fired between stop() and IsLive=false saw IsLive=true and called start(obj.LiveTimer) on a freshly-deleted timer, triggering MATLAB's own "Error while evaluating TimerFcn" loop. Fix: flip the order (IsLive=false first so the ErrorFcn can't restart) and guard stop/delete with isvalid + try/catch — same pattern LiveTagPipeline.stop() already uses. Out of scope for Phase 1012 scope, but blocks CI. Zero behaviour change in happy paths; only affects teardown-during-error-loop.
…CI loop The previous version set TimerFcn to always throw, then pause(0.5) to observe the ErrorFcn restart. On MATLAB CI the error-then-restart loop outpaced teardown, producing ~500k stderr lines and hanging the Tests workflow (PR #59 observation). New version uses a one-shot TimerFcn (first tick errors, subsequent ticks no-op) backed by a containers.Map counter (handle class — mutates across anonymous-function invocations). Verifies the same property: ErrorFcn restarts the timer after a TimerFcn throw. Combined with the earlier stopLive IsLive-first fix, teardown is now race-free.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
pause(0.3) was unreliable inside matlab.unittest — the harness services
timer callbacks differently than top-level scripts, sometimes not
within the pause window. Replaced with bounded polling (`while
counter('n') == 0 && toc < 3.0; pause(0.05); end`) which is robust
across runtime environments.
Verified locally: test passes in 1.2s on MATLAB R2025b. Full
TestDashboardEngine suite: 18 pass + 1 pre-existing failure
(testAddWidgetWithTag uses deleted Threshold class, broken since
Phase 1011 — not introduced by this PR).
testErrorInvalidWriteMode called writeTagMat_ directly from tests/suite/, which MATLAB's private-folder scoping blocks. Added a 'write' dispatch case to the existing shim (same Major-1 Option A pattern used by parse/sniff/select) and routed the test through it. Verified locally on MATLAB R2025b — test passes in 0.22s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1012 delivers the Tag ingestion pipeline — two new classes under
libs/SensorThreshold/that read arbitrary delimited raw files (.csv/.txt/.dat) and emit per-tag.matfiles keyed offTagRegistry:BatchTagPipeline— synchronous orchestrator. IteratesTagRegistry, de-dups file reads, ingests each eligible tag, throwsTagPipeline:ingestFailedwith a failure report at end-of-run.LiveTagPipeline— timer-driven orchestrator. MirrorsMatFileDataSource'smodTime + lastIndexpattern on raw text files; appends new rows viaload → concat → save(notsave('-append')).SensorTag.RawSource+StateTag.RawSource— new NV-pair struct property(file, column, format)that binds a tag to its raw source. Driven off existing sensor-extras / state-extras splitArgs machinery.Tag.mbase class is untouched (Pitfall-1 discipline preserved).readRawDelimited_,selectTimeAndValue_,writeTagMat_) underlibs/SensorThreshold/private/— one delimited-text parser handles CSV/TXT/DAT via auto-detected delimiter; shape dispatcher supports both wide (time + N value cols) and tall (2-col) layouts.readRawDelimitedForTest_) — public dispatcher explicitly marked test-only, routingparse/sniff/selectto the private helpers fromtests/suite/(needed because MATLAB's private-folder scoping blocks direct test calls). Grep gates in both pipeline classes enforce zero production calls.Context
SensorTag.load()callers; the pipeline's output.matsatisfies that contract unchanged.MonitorTagandCompositeTagare never materialized to disk by the pipeline (D-16). TheirgetXY()stays lazy at plot/dashboard load time, preserving MONITOR-03..planning/phases/1012-…/for reviewer reference.Coverage
TagPipeline:*error IDs both emitted and asserted by test methods.matlab.unittestsuite).Notable flags (non-blocking, documented)
TestSensorTag.m+TestStateTag.m— were edited additively in Plan 02 but not explicitly counted in the file ledger). Substantive tests; no functional impact.@ClassName.staticPrivatehandles and the documented fallback;BatchTagPipeline.eligibleTags_uses a private-method handle pattern that ships correctly on MATLAB but falls over on Octave — logged in 1012-.../deferred-items.md.LiveTagPipelineuses the inline-lambda workaround, proving the fix is trivial when we come back to it. Does not block MATLAB usage.Wave / commit structure
Commits trace the planned wave execution so the history reads like the plan — easier review:
1dfde95) — fixture helper + RED suite scaffolds (47 placeholders, TDD foundation).7de5f3c,236ba01) —RawSourceNV-pair + private parser/writer helpers + test shim. Parallel execution.6c3e156,480765d) —BatchTagPipelineclass, two-commit checkpoint (skeleton, thenrun()body + GREEN suite).1ae70fc) —LiveTagPipelineclass + 11 GREEN tests.cf3b713) → roadmap + STATE (53e3bf8) → PROJECT.md evolve (d17e3dc).Test plan
tests/run_all_tests.munder MATLAB R2020b+ — confirm all 75 pre-existing tests stay green and newTestBatchTagPipeline(18) +TestLiveTagPipeline(11) +TestRawDelimitedParser(18) suites pass.tests/run_all_tests.munder Octave 7+ — confirm the same. Note theLastFileParseCountpath inBatchTagPipeline.eligibleTags_may hit the deferred Octave-parity defect;LiveTagPipeline+ the full parser/writer paths are unaffected.mh_style libs/SensorThreshold tests/suite.SensorTagwithRawSource, runBatchTagPipeline, callSensorTag.load()on the emitted.mat, confirmX/Ymatch the source CSV.🤖 Generated with Claude Code