Subsample instead of dropping frames with too many star candidates; improve star extraction robustness

[Cybis320]

Motivation

Data from the MQTT health monitor dashboard shows that many stations regularly drop FF files from star extraction. The drops are disproportionately the best FF files of the night, since clear, dark skies produce the most star candidates and trip the overflow limit.

Problem

When the number of star candidates on a frame exceeds max_stars, extractStars() drops the entire frame ("Too many candidate stars to process!") and returns zero stars. This triggers in two very different situations:

• Genuinely star-rich frames — a sensitive camera under a dark, transparent sky legitimately detects more candidates than the cap. The all-or-nothing gate discards exactly the best frames of the night, and the per-FF star count flips between high values and zero.
• Bright / noisy frames — the candidate test (local max − min > threshold) passes on steep brightness gradients (moon glare, cloud edges) and on sensor noise, flooding the candidate pool on high-gain cameras and moonlit nights. On real archived data this wipes out large parts of nights: one archived night from a high-gain camera has only 189 FF entries in CALSTARS out of a full night; an adjacent night from the same camera lost essentially everything.

A side effect of the old behavior: intensity_threshold had to be tuned high enough to keep candidate counts under max_stars, or frames would be dropped — so sensitivity and processing cost could not be set independently.

Two related issues:

• The max_global_intensity gate discards frames silently (no log trace, indistinguishable from a genuinely starless frame), and its default of 150 rejects frames that measurably still hold dozens of stars.
• In extractStarsImgHandle (video/image-dir processing), a single too-bright chunk aborts all chunks, discarding stars already extracted and disabling meteor detection for the whole file.

Changes

• Candidate overflow no longer drops the frame. The candidates are subsampled: ranked by white top-hat prominence (image minus its morphological opening — insensitive to gradients, so stars rank above glare noise) and selected on a spatial grid with an equal quota per tile, leftovers filled by global prominence. On star-rich frames this yields a spatially uniform sample across the full FOV (what the distortion fit and recalibration want) rather than a brightest-first, center-biased one; on flooded frames it bounds junk to the affected region's tiles. The PSF fit vets the sample as before. Frames under the limit are untouched — output is bit-identical.
• The tuning knobs are now independent: intensity_threshold can be set purely for desired sensitivity and max_stars purely as the compute budget — a low threshold no longer risks dropping frames, it just engages the subsampler.
• max_global_intensity gate is logged at both call sites (median and cutoff values), and the default is raised 150 → 230. The gate operates on the flat-corrected median; frames at median 190–225 were measured to still hold 20–45 stars with 75–95% frame-to-frame repeatability.
• Too-bright chunks are skipped, not fatal, in extractStarsImgHandle — other chunks keep their stars.
• SkyFit2 Star Detection tab:
  • New Max Global Intensity slider.
  • The max-stars budget is split into a session-only SkyFit Max Stars (deep detection for initial calibration, never saved) and Config Max Stars (what Save Config writes; 400 recommended for the nightly pipeline).
  • New Reset to Defaults button.
  • Help topic updated.

Note

extractStars() no longer returns False; the callers' status == False branches remain but are now dead paths.

Validation (real FF files, several stations)

Test	Result
Regression: 160 archived FFs, old vs. new code	Identical star lists on every frame (overflow path never triggers on normal frames)
Recovery on production-dropped flood frames (replayed with the exact station config: mask only, no flat, threshold 18, max_stars 400)	331 PSF-vetted stars in ~1 s where CALSTARS recorded nothing
Star quality, measured offline via repeatability: extraction run independently on consecutive 10 s FFs, stars matched within 2 px (real stars drift <1 px in 10 s and match; noise doesn't — chance rate <1%)	Prominence-selected samples repeat at 92–98%, better than the 76–89% of the uncapped pipeline
Stratified vs. pure global ranking, binding cap, no flat (the common network configuration)	Retains 60/93 baseline stars vs. 25/93 for global ranking on a gradient-contaminated frame; ties on clean frames
Compute	Unchanged on normal frames; overflow frames add one separable max-filter pass plus PSF fits capped at max_stars — same worst case as a legitimately rich frame today

Science impact

Per-star measurements are unchanged (same PSF fit). Junk candidates can't match the star catalog, so recalibration, photometry, and flux cloud detection are structurally unaffected. The net effect is additive: frames that previously contributed zero stars now contribute vetted samples — including the best dark-sky frames on sensitive cameras — so expect a completeness step at deployment (more usable observing time in long time series).

[dvida]

Thanks Luc — this is a well-motivated and carefully validated PR. I did an in-depth review of the diff plus an independent empirical check of the core algorithm, and audited every downstream CALSTARS consumer. Summary below so you know where to pick things up.

What I verified (independent tests, synthetic star fields, old vs. new code)

• Bit-identical under the cap — confirmed. 149 candidates, cap 1000: all 8 output arrays identical between old and new code.
• Flood recovery — confirmed. Moon-glare + heavy noise frame, 990 candidates, cap 200: old code returns False (frame lost), new code recovers 132/134 true stars outside the flooded region, with only ~17 junk detections surviving the PSF fit.
• Spatial uniformity — confirmed. 600-star clean field, cap 200: 4×4 grid occupancy 9–17 per cell, 200/200 detections match injected truth.
• Edge cases — no crashes at cap = candidates−1, very small caps (quota-1 path), or a single-region flood.
• Algorithm internals — maximum_filter(minimum_filter(img)) is a true grayscale opening so the top-hat prominence is correct; label indexing has no off-by-one; the quota loop preserves prominence order in the leftovers and selects exactly max_star_candidates; the extra maximum_filter pass only runs in the overflow branch, as claimed.

The subsampler itself is sound — I could not break it. The items below are all in the surrounding GUI/config plumbing and in the framing of downstream effects.

Requested changes

1. Reset to Defaults resets camera gamma to 1.0 — can silently corrupt the station config and live photometry.
resetToDefaults() (CustomPyqtgraphClasses.py) loops over all slider defaults, including gamma (100 → 1.0). Gamma is a hardware property (0.45 on many IMX cameras), not a tuning preference. Failure path: user clicks Reset to fix detection sliders → updateGamma(1.0) fires → with override enabled it immediately rewrites platepar.gamma and config.gamma in memory → Save Config now writes [Capture] gamma = 1.00 (new in this PR). Photometry breaks with no warning. Suggested fix: exclude gamma from the reset, or reset it to the station's config value rather than the global default. Arguably Config Max Stars should also reset to the configured value, not 400.

2. The config max-stars slider snaps/clamps, then Save Config writes the altered value.
The slider has range 100–2000 with snap-to-100, and loadFromConfig() seeds it from config.max_stars. A station config with max_stars: 150 shows 200 on open, isConfigModified() returns True with zero user input (Save button enabled at startup), and saving writes 200 back. Values below 100 or above 2000 get clamped the same way. Suggested fix: seed the slider without snapping (snap only user motion), or track a "user touched" flag so the seed value never counts as a modification.

Points for discussion (no code change strictly required)

3. "Science impact: structurally unaffected" holds for catalog-matched paths, but several consumers act on raw CALSTARS counts and do react to junk entries.
Protected, as the PR states: astrometry fit acceptance (matchStarsResiduals, recalibrateFF vetoes) and the flux clear-sky ratio (Utils/Flux.py counts catalog-matched stars) — junk cannot corrupt platepars or cloud detection. Exposed:

• The meteor-detection gate len(star_list) >= ff_min_stars (DetectStarsAndMeteors.py:235): flooded moonlit frames that used to yield 0 stars now pass, so meteor detection runs on noise frames. Partly the point of the PR, but expect some extra false-positive detections in FTPdetectinfo on bad nights.
• autoCheckFit picks its NN-alignment seed as the FF with the most raw stars (CheckFit.py:508) — a junk-flooded frame can win and seed the alignment badly. Cheap fix: pick by matched or SNR-filtered count.
• The flux sensor FWHM is a median over all raw CALSTARS entries (Utils/Flux.py:1821) — in my flood test ~8% of surviving detections were junk, enough to bias the median on bad nights. Cheap fix: SNR-filter before the median.
• CalibrationReport selects its representative frame by raw count; calstars_min_stars-type total gates inflate similarly. Cosmetic.

4. Default rollout reach. The ConfigReader.py (150→230) and .config template changes only affect installs whose config file lacks max_global_intensity — essentially all deployed stations have it written in their own .config and will keep 140/150 until an operator edits it. So the measured benefit at median 190–225 mostly won't reach the existing network through this PR alone. Worth a note in the PR text on whether that's the intent.

Minor

• Docstrings at ExtractStars.py:320 and :430 still say "maximum mean intensity" — it's the median now (and the code default is 230).
• The dead status is False branches are acknowledged in the PR and harmless; fine either way.
• GUI plumbing all checks out on close read: _original_config_gamma is initialized and None-guarded correctly, every former tab.indexOf() call site was converted to tabIndexOf(), the Auto Levels button and Ctrl+A stay in sync, and the snap handlers' setValue re-entry terminates after one pass.
• The scroll-area wrappers / Auto Levels button / hidden info panel are unrelated UI improvements riding along in a star-detection PR — fine for prerelease, just makes bisecting a bit harder.

Overall: the core change is a clear improvement and the validation in the PR description reproduces. Once items 1 and 2 are addressed (both small fixes), this is good to go from my side.

[Cybis320]

Thanks for the deep review, Denis, and especially for the independent empirical checks - the flood-recovery and uniformity verifications are exactly the tests I'd have wanted a second pair of eyes on.

Both requested changes are addressed, plus the cheap fixes from the discussion items:

1. Gamma / Reset to Defaults - fixed. Reset to Defaults now returns the station-bound values to the loaded config instead of global defaults: gamma goes back to the station's configured value (it's a hardware property, as you said), and the config max stars likewise. Only the tuning sliders reset to the recommended defaults.

2. Slider snap on load - fixed. Programmatic seeding now bypasses the snap-to-100 handlers, so max_stars: 150 loads and displays as exactly 150, is not flagged as an unsaved modification, and Save Config writes it back unchanged. User motion still snaps. Out-of-range config values extend the slider range instead of being clamped.

3. Raw-count consumers - measured both suggestions before/after; one is in, one turned out to be counterproductive:

• autoCheckFit now seeds its NN alignment from the FF with the most SNR-filtered stars (SNR >= 5), falling back to the raw count for older CALSTARS files that pad the SNR column with -1. Measured on a junk-flooded night: the raw-count winner was a frame where only 6% of detections match the catalog; the SNR-count winner matches 15% with more genuine stars. On clean nights the two rankings pick equivalent frames.
• The flux FWHM median: I implemented the SNR filter, then measured it - and reverted it. On a frame that is 94% junk the raw median differs from the matched-stars-only median by just 0.07 px: junk that survives the PSF fit has star-like FWHM by construction (it passed the shape checks), so the median is robust to it exactly as designed. The filter, however, biases the median by +0.4 px on perfectly normal nights, because standard-settings detections have median SNR ~3.5 and an SNR >= 5 cut keeps only the brightest third. The cure measurably exceeds the disease, so the median stays unfiltered (with a comment recording the measurement).
• The meteor-detection gate passing flooded frames is indeed partly the point - those frames can contain real meteors that were previously unfindable. Agreed that some extra false positives on bad nights is the trade-off; they still have to survive the detection pipeline itself, so the exposure is bounded.
• One addition to your consumer audit: the flux clear-sky ratio is junk-proof as you said, but it has the opposite exposure - its numerator (catalog-matched detections) is bounded by max_stars while the predicted denominator is not. On star-rich nights with sensitive cameras the prediction can exceed the cap severalfold (detector LM ~8 puts ~900 predicted stars in a 40x22 deg FOV against a 400 cap), so a perfectly clear sky would read as cloudy. This predates the PR - such frames used to be dropped outright and always read as cloudy - but now that they deliver a full cap of matches it's worth fixing properly: the prediction is now saturated at min(predicted, config.max_stars), so the ratio compares against what detection is actually allowed to deliver.

4. Rollout reach - correct. Deployed stations keep their configured max_global_intensity until an operator changes it; this PR raises the default for new installs and gives operators the recommendation plus the SkyFit UI to apply it deliberately. It's the usual catch 22 with our config scheme - we could retire and rename the config field if we later think it's needed?

Docstrings at ExtractStars.py fixed (median, default 230).

Pushed as four commits on top of the branch (including the measured revert of the flux FWHM filter and the clear-sky prediction saturation).