Forecasting With a Known Answer: A Deductive-Ground-Truth Case Study on the Clock-Killing First Down
Working draft; the title is not final.
Forecasting evaluation almost never knows the right answer. A model is scored against a realized value that was one draw from an uncertain process, so a well-calibrated belief and a lucky one are hard to tell apart. We study a setting where the answer is known by arithmetic. In-game National Football League play-by-play contains a deductive end-state, the clock-killing first down: a conversion by the leading team late enough that it can kneel out the remaining clock given the trailing team's timeouts. The outcome is fixed, and we label it from raw game state without any expected-points or win-probability model, which gives a forecast evaluation a known answer against which recovery is a measurement rather than an inference.
A series-only estimator must miss this certainty for a model-independent reason: deductive certainty is not a function of the win-probability series, so any estimator built on that series falls short of the deductive outcome however well calibrated it is. The existence of that shortfall is the general claim; its size, about 0.03 for the production model we use, is incidental and model-specific. The question a known answer lets us ask is who recovers the shortfall, and we find one clean result: no rule-free signal closes the deductive cell. A gradient-boosted model, a multilayer perceptron, and a foundation model fed the clock as a covariate all leave the shortfall; the cell is a tenth of a percent of any rule-free region and pays no rent in average loss, so no loss-minimizing learner finds it. Upweighting an observable late-game region lifts the cell only partway and plateaus short of the answer, and does so partly as blanket region-confidence rather than by localizing the cell. The boundary is easy to fit once it is found: upweighting the deductive label localizes the cell, and the model then learns the boundary and generalizes it to held-out events. The obstacle is localization, not representation or generalization, and only label or rule information closes the cell. We contribute the paradigm of deductive ground truth for forecast evaluation, a model-independent deductive label, and this localization result, as a case study on one event.
A forecasting benchmark scores a prediction against a realized outcome, and the realized outcome is usually one sample from a distribution the forecaster was right to be uncertain about. A model that said 0.7 and a model that said 0.9 both look correct when the event happens, and the data cannot say which belief was better without many repetitions. The ground truth is stochastic, and aggregate scores work around that rather than resolving it.
Some end-states in sports are not stochastic. The clock-killing first down is a conversion by the leading team, late enough to kneel out the remaining clock given the trailing team's timeouts and the play clock. Whether it can is an arithmetic identity over observable state, and once it holds the game is over. We label the event from the clock, the score, and the timeout counts, without using the expected-points or win-probability models, so the label is independent of the quantities a forecaster predicts.
A known answer turns a familiar limit into a measurement, for a reason that does not depend on the model under test. Deductive certainty at the clock-killing first down is not a function of the win-probability series, so any series-only estimator falls short of the deductive outcome no matter how well calibrated it is. The production model gives the leading team about 0.97 where the clock makes the outcome 1; the existence of that shortfall is model-independent, and its size, about 0.03, is incidental to one model's calibration and would shrink under a sharper one. Because the answer is known we can ask who recovers the shortfall, a question sharper than a benchmark can usually pose.
The result is a localization claim, and it is sharp. The whole univariate family leaves the shortfall: the feature-free win probability, three zero-shot foundation models, and a random walk, all near 0.97, which on a martingale target is the current value by construction. No rule-free signal closes it. A production win-probability model that already conditions on the clock leaves it, a gradient-boosted model trained on the clock features leaves it, that model handed the exact kneel-out margin leaves it, a multilayer perceptron leaves it, and a covariate foundation model fed the clock and its known future values leaves it. Emphasizing an observable late-game region during training lifts the cell only partway and plateaus short of the answer, partly as blanket region-confidence rather than localization. Across three model families, a learner that minimizes average loss does not localize a cell that is a tenth of a percent of any rule-free region, because the cell pays no rent in the average loss. The boundary is not hard to fit once it is found: upweighting the deductive label localizes the cell, and the model then learns the boundary and generalizes it cleanly to held-out events, reaching 0.997 on clock-kills whose labels it never saw. The obstacle is localization, not representation or generalization, and only label or rule information closes the cell.
The contribution is three things. The paradigm of deductive ground truth for forecast evaluation, which lets recovery be a measurement rather than an inference, and which is the part that could carry beyond the sports setting. A deterministic, model-independent labeler for the clock-killing first down. And a clean result on it: a characterized residual, and a demonstration that data-driven methods do not recover it while label or rule information does. We offer this as a case study on a single deductive event, not as a benchmark suite; the two-axis structure of football leverage that locates the clock-kill is background, in Appendix A.
Zero-shot time-series foundation models forecast new series without fitting to them. We use three as the univariate and covariate rungs of the recovery ladder rather than as the subject; on a martingale target they reduce to the current value, and the paradigm and the result do not depend on which model is current. Work on the calibration of these models reports they are usually well calibrated in the coverage sense, which is consistent with our finding that no standard calibration diagnostic flags the break.
The econometric literature on structural breaks detects and dates breaks rather than forecasting through them; our break is known and its outcome is fixed, so the question is recovery, not detection. The nearest prior work studies sequential probability forecasts on NFL and NBA win-probability paths against a martingale benchmark and uses no foundation model; it also reports, on independent data, that NFL win probability is martingale-consistent, which agrees with our verification. We are not aware of work that uses a deductively labeled in-game end-state as forecast ground truth.
We use nflfastR play-by-play for the 2010 through 2024 regular seasons through nfl_data_py. After keeping regular-season plays, removing overtime and non-plays, and requiring defined expected-points and win-probability changes, the sample has 595,510 plays across 3,903 games. Expected points and win probability are nflfastR model outputs, used as a target and a reference.
Home win probability is a clean martingale here, which we verify rather than assume: the one-step change has mean near zero, the regression of the next value on the current value has slope 0.998 and intercept 0.001, and the mean change is within 0.002 of zero at every win-probability decile. A consequence we use is that a univariate nowcast equals the current value, so a foundation model reduces to the feature-free estimate at the break, which we then confirm.
A first down by the leading team, late enough to kneel out the remaining clock given the trailing team's timeouts and the play clock, ends the game. We encode the kneel-out arithmetic over the clock, the timeout counts, and the play clock, and validate it against known kneel-out sequences. The rule uses only raw game state and not the expected-points or win-probability models.
Across the fifteen seasons there are 504 clock-killing first downs, 263 in one-score games. We evaluate out of sample with a walk-forward backtest: each season from 2013 on is scored by models trained only on earlier seasons, and we pool the out-of-sample clock-kills, 398 in all and 208 of them one-score. The leading team won all 398. The determinism is historical and close to but not exactly one, since victory-formation fumbles and defensive scores have occurred; the labeler's timeout arithmetic is what holds the realized rate at one, correctly excluding boundary cases where the trailing team still holds a timeout and the kneel-out is not yet decided. We treat the label as the near-certain outcome it is rather than an absolute guarantee.
The clock-kill is small on both named leverage axes. In one-score games the leading team's win probability before the play has a median near 0.964, the conversion's win-probability change a median near 0.017, and its expected-points change a median near 0.835. The certainty the play carries does not appear as size on either axis; it lives in the clock state. The residual this implies has two parts that differ in generality. That a positive residual exists for any series-only estimator is model-independent, since deductive certainty is not a function of the features the series carries, so any such estimator must fall short of 1 at a deductively decided state. The size near 0.03 is specific to nflfastR, whose win-probability model smooths at the extremes to limit overconfidence, so a more sharply calibrated model might read 0.98 there and shrink it. Our claims concern the existence of the residual and its recoverability, not its magnitude.
Figure 1. The two-axis leverage space on 2010-2024, absolute win-probability revision (|wpa|) against absolute expected-points revision (|EPA|), shaded by play density on a log scale. The two axes are correlated but distinct, Pearson 0.640, and the high-|wpa|, low-|EPA| corner above the dashed ninetieth-percentile line is populated. The 504 clock-killing first downs, red for the 263 in one-score games and orange for the 241 with nine-or-more-point leads, sit low on the win-probability axis, the one-score conversions at a median |wpa| near 0.017, so the certainty they carry does not appear as magnitude on either leverage axis and is read instead from the clock state. This is why the study carries the event on the deductive label rather than on win-probability movement.
The unit is the play. With the outcome known, we evaluate predictors of the probability assigned to the leading team, which wins on the test clock-kills, as a nowcast at the clock-kill state. Because every test clock-kill resolves the same way, this is a degenerate-outcome set, so the informative quantity is the probability a method assigns, not its accuracy, and the whole burden of separating a justified high probability from generic leader-favoring confidence falls on a calibration check in the adjacent cell, described below. We climb a ladder of how much clock information a method has, training where needed on all earlier seasons in a walk-forward backtest and evaluating on the 398 pooled out-of-sample clock-kills with a game-clustered bootstrap:
- the feature-free win probability, a production model that already conditions on the clock and timeouts;
- three zero-shot foundation models, Chronos-Bolt, TimesFM-2.5, and TiRex, and a random walk, each forecasting the win-probability series one step from a context ending at the clock-kill, and Chronos-2 both univariate and with the clock and timeout state as exogenous covariates including their known future values;
- a gradient-boosted classifier on the raw clock features, the same with the exact kneel-out margin as a feature, the same with an observable late-game region upweighted, and the same with the deductive clock-kill label upweighted;
- a deductive oracle that applies the kneel-out arithmetic to observed state.
The two upweightings are the crux, and we keep them distinct. The observable region is fourth-quarter plays with the leading team in possession and under 180 game-seconds, which requires no kneel-out arithmetic and contains all 398 out-of-sample clock-kills as a small subset. The label upweighting uses the deductive clock-kill label itself, so it consumes the answer. We audit both with a calibration check on the adjacent near-miss cell, the 7,860 late-game leading plays in the out-of-sample seasons that are not clock-kills, where the leading team does not always win, which is where label-derived overconfidence would show; and with a sweep over the weight multiplier, so the recovery is not reported at a single tuned point. A fourth foundation model, Moirai-2, did not install in our environment.
Each predictor outputs a probability for the leading team. The table reports the mean assigned probability with a game-clustered interval and the residual on the 398 out-of-sample clock-kills in 390 games, grouped by how much clock information the method has.
| method | P(leading team) | 95% CI | residual |
|---|---|---|---|
| univariate (no clock beyond the series) | |||
| feature-free win probability | 0.973 | [0.969, 0.977] | 0.027 |
| random walk | 0.973 | [0.969, 0.977] | 0.027 |
| Chronos-Bolt | 0.973 | [0.969, 0.977] | 0.027 |
| TimesFM-2.5 | 0.968 | [0.963, 0.972] | 0.032 |
| TiRex | 0.972 | [0.968, 0.976] | 0.028 |
| Chronos-2 (univariate) | 0.969 | [0.964, 0.973] | 0.031 |
| covariate access (the clock as input) | |||
| GBM, raw clock covariates | 0.967 | [0.962, 0.972] | 0.033 |
| MLP, clock features | 0.958 | [0.950, 0.964] | 0.042 |
| GBM, exact kneel margin | 0.969 | [0.964, 0.973] | 0.031 |
| Chronos-2, clock covariates | 0.971 | [0.967, 0.975] | 0.029 |
| observable-region emphasis (no label) | |||
| GBM, late-game region upweighted | 0.983 | [0.979, 0.987] | 0.017 |
| label or rule consumed | |||
| GBM, deductive label upweighted | 0.993 | [0.992, 0.994] | 0.007 |
| oracle (deductive rule) | 1.000 | [1.000, 1.000] | 0.000 |
The univariate family is one rung at about 0.97, every interval overlapping, the foundation models reducing to the current value with no series-end deviation. Covariate access does not lift off it. The production win-probability model conditions on the clock and leaves the residual, the gradient-boosted model on the raw clock features leaves it at 0.967, and Chronos-2 fed the clock as an exogenous covariate leaves it at 0.971, indistinguishable from its own univariate forecast. The sharpest case is the kneel-out margin. We hand the gradient-boosted model the exact quantity the rule thresholds on, so the deductive boundary collapses to a single axis-aligned split at margin zero, and the tree still leaves the residual at 0.969. The model has the exact discriminating feature and cannot use it, which rules out feature availability and representation as the obstacle and isolates the mechanism as loss geometry and rarity: a cell that is a tenth of a percent of the data pays no rent in the average loss, so no method that minimizes average loss carves it out, whether the clock arrives as a feature or a covariate. The same holds across model families: a multilayer perceptron on the clock features leaves the residual at 0.958, with a comparable held-out win-probability AUC to the tree, so the failure is the loss geometry and not one architecture.
Emphasizing an observable region recovers only part of it, and not by localizing the cell. Upweighting fourth-quarter leading plays under 180 seconds, a region defined without the kneel-out rule, lifts the clock-kills from 0.973 to 0.983, and the weight sweep plateaus there, 0.981 at weight ten and 0.983 at weight three hundred, well short of the deductive answer. The lift is largely blanket region-confidence rather than discovery of the boundary: the same upweighting raises the adjacent near-miss cell to a mean 0.965 against an actual 0.957, so the model leans on the whole late-game neighborhood rather than carving out the deductive cell. Reweighting toward the region approximates restricting training to it, where the cell is about five percent of plays rather than a tenth of a percent, so this also answers whether concentrating on the late-game region would help: it moves toward the region's average, not to the deductive answer.
Only consuming the answer recovers it. Upweighting the deductive clock-kill label reaches 0.993, and the weight sweep shows the recovery rising monotonically with the weight, 0.978 at ten and 0.997 at three hundred. The model is not failing to generalize. At weight three hundred it assigns 0.997 to the 398 out-of-sample clock-kills whose labels it never saw, so the boundary is learnable and generalizes cleanly across the walk-forward folds once the label points at the cell. What it cannot do without the label is find the cell, and the obstacle we isolate is localization, not representation or generalization. This recovery is specific to the labeled cell rather than blanket confidence toward the leader. On the 7,860 near-miss plays of the out-of-sample seasons the label-upweighted model assigns the leading team a mean 0.972 against an actual 0.957, a mild inflation, and binned by predicted probability it tracks the truth in the tails, 0.821 against an actual 0.818 in its low bin and 0.997 against an actual 0.991 in its high bin, with the overshoot concentrated in the middle band rather than spread uniformly. The recovered clock-kills sit in the calibrated high bin, but they do not come for free in the neighborhood. On the 855 plays of the adjacent 0.95 to 0.98 band the label-fitted boundary overshoots, 0.968 predicted against 0.909 actual, a 5.9-point overconfidence larger than the residual itself and larger than the two points the upweighting recovers on the cell. The overshoot does not reach the recovered cell, and it is not a defect to apologize for but the smoothing bias seen from the other side: even when the label points the learner exactly at the cell, the learner cannot represent the sharp arithmetic threshold without smearing onto its nearest neighbors, which is the same bias that left the cell undiscoverable to begin with. For the claim that only consuming the label localizes the cell, the bleed is beside the point and if anything supports it. We are not proposing label upweighting as a technique, since it trades a two-point recovery on the cell for a six-point bleed onto its neighbors; we are showing that no rule-free signal localizes the cell and that even label-localized fitting carries the smoothing bias the paper is about. The recovery still consumes the label that defines the outcome, which is information about the answer, not about modeling. The oracle removes the residual exactly, by applying the rule, and sits at the top of the ladder as the deductive bound.
The reading is a spectrum of how much label or rule information a method incorporates. No rule-free signal closes the residual from observables, and observable-region emphasis lifts it only partway. Label-informed emphasis recovers most of it, and the rule recovers it exactly. The information is present in observed state, since the oracle reaches it from the same fields, but no rule-free signal localizes the cell at this rarity, since nothing that minimizes average loss without the label reaches it. The cell is invisible to every rule-free localization signal. It is low-loss rather than hard, since the win-probability model already says about 0.97 and the leading team wins, so the per-event loss is about 0.03; difficulty-based reweighting, focal loss or hard-example mining, would no more surface it than region reweighting did. Only the arithmetic rule separates the cell from its neighbors, so the localization has to be supplied.
Point error and coverage do not separate the clock-kill from win-probability-matched controls, and the probability integral transform shows no boundary pile-up for any model: at the clock-kills the per-model PIT means sit near 0.5, between 0.46 and 0.49, with at most six percent of the mass above 0.9, so the forecasts concentrate near the center rather than piling at the boundary, marking intervals that are if anything too wide for how little the win probability moves. The model is conservative at the clock-kill, not overconfident, and the realized value sits inside its interval, so there is no operational calibration failure. The residual is visible only against the known answer. Figure 2 shows the transform, and the detailed calibration probe is in Appendix B.
Figure 2. Probability integral transform of the win-probability forecast at clock-kills (red) against win-probability-matched controls (blue, capped at 1,500), for three foundation models and the random-walk floor. A calibrated forecaster gives a flat histogram at density one. At the clock-kills the mass concentrates near the center rather than piling at zero or one — per-model PIT means 0.46 to 0.49, at most six percent of the mass above 0.9 — marking intervals that are if anything too wide for how little the win probability moves. No panel shows the boundary pile-up that would signal overconfidence, so no standard calibration diagnostic flags the decided state.
A known answer is the contribution. Forecast evaluation scores against stochastic outcomes, so it cannot cleanly ask whether a method recovered a specific quantity. A deductive end-state can. The clock-killing first down gives an outcome fixed by arithmetic and labeled without the models under test, against which recovery is a measurement. That paradigm is what we contribute, and the result it produces is the spectrum in Section 5.1.
No rule-free signal closes the residual, and that is one mechanism, not several surprises. A production win-probability model, a gradient-boosted model, a multilayer perceptron, and a covariate foundation model all leave the residual, and observable-region emphasis lifts it only partway, plateauing short of the answer as blanket region-confidence. This is what minimizing average loss predicts: a 0.1 percent deductive cell is averaged into the surrounding population, whether the clock arrives as a feature, a covariate, or an upweighted region. The production model uses the clock for the average relationship, not the threshold, and that is the right reason to have expected covariate access to fail rather than to be surprised it did. We present these as a robustness result spanning three model families: the production win-probability model, the gradient-boosted model, and the region-upweighted model are configurations of one gradient-boosted family, the covariate foundation model is a transformer, and the multilayer perceptron is a third, so the mechanism holds across distinct architectures rather than within one. A method with a threshold inductive bias, a rule-search or symbolic learner, would supply exactly the structure we argue must be supplied, so its success would confirm the account rather than contradict it.
Recovery consumes the answer, and we say which form. What recovers the residual is information about the answer. The oracle is the rule at inference. The label-upweighted model is made to attend to the labeled cell through sample weights, which is the deductive label laundered into the training objective; it recovers most of the residual, with a boundary overshoot on the nearest neighbors that is the same smoothing bias seen from the other side, and it is not a model that discovered the structure. The honest statement is a spectrum from no clock information, through covariate access that does not recover and observable-region emphasis that recovers only partway, to label-informed emphasis that recovers most and the rule that recovers all. The paradigm is what makes that spectrum measurable.
Scope. This is one deductive event with 398 out-of-sample test cases in one league. We call it a case study and a proof of concept for deductive-ground-truth evaluation, not a benchmark, because a benchmark implies a suite that discriminates across tasks and we have depth on a single event. Whether the paradigm generalizes is the natural next step: other deductive end-states, in this sport and in other clock sports, labeled the same way, would make it a suite. The residual is small in absolute terms, the recovery by emphasis is shown on one learner family, and the determinism is near-certain rather than guaranteed.
Toward a suite. The paradigm asks only for an end-state whose outcome is fixed by an arithmetic over observable state and can be labeled without the model under test, and several exist. In basketball, a trailing team that cannot tie or take the lead within the possessions the clock, the shot clock, and the foul situation allow is in a decided game, labelable from score, clock, team fouls, and timeouts. In fixed income, a callable bond past its call date once a rate condition holds has a deductive price to call, labelable from the rate path and the call schedule. In operations, a queue with arrivals closed and a fixed service rate has a deductive drain time, labelable from the queue length and the rate. Each supplies what the clock-kill does, a known answer at a labeled point, and each would test whether the failure to learn the boundary that we find is particular to this cell or general to sharp deductive structure in a rare region.
Reproducibility and artifact. The harness is the reusable part: deterministic, play-indexed windowing, the clock-killing labeler, the recovery ladder with the proxy and label upweightings and the near-miss audit, and the oracle. The code and the deterministic labeler are released at https://github.com/CanadaApollo6/deductive-eval under a permissive license, with fixed seeds and deterministic window generation. The data are pulled and cached from nflfastR under its terms. Zero-shot inference for the foundation models runs on a single consumer GPU; the labeler, the gradient-boosted models, and the audit run on CPU.
When a forecasting target has a deductive end-state, evaluation can ask who recovers a known quantity rather than who scores well against a noisy one. On the clock-killing first down the win-probability series leaves a small, exactly characterized residual that the whole univariate family leaves alike. No rule-free signal closes it: covariate access does not recover, and observable-region emphasis lifts it only partway; the residual is closed only by consuming the deductive label or applying the rule. The boundary is learnable once a label localizes the cell, so the obstacle is localization and not representation or generalization. The information is present in observed state, since the rule reaches it, but no rule-free signal localizes it at this rarity, and the deductive-ground-truth paradigm is what lets that be measured rather than argued. We offer it as a case study and a reusable artifact, with a suite of deductive end-states left as the next step.
Leverage in football has two correlated but distinct axes, the absolute per-play change in expected points (|EPA|) and the absolute change in win probability (|wpa|), with Pearson correlation 0.640 and Spearman 0.581, and a populated high-|wpa|, low-|EPA| region of about 4,060 plays at the ninetieth percentile of |wpa| (about one per game). This formalizes a distinction familiar in football analytics and locates the clock-kill as small on both axes. Over the 66,539 break windows at the ninetieth percentile of |wpa|, expected-points forecast error correlates with |EPA| far more than with |wpa| for all four predictors — Spearman near 0.35 against |EPA| and near 0.07 against |wpa| — with game-clustered intervals that cleanly separate the two, |EPA| within about [0.34, 0.37] and |wpa| within about [0.06, 0.10]. This is a property of the series, since the random walk reproduces it, and it does not bear on where the win-probability residual appears at the clock-kill; we keep it as setup. We do not test an expected-points-aware model at the clock-kill, so the smallness on the |EPA| axis is a locating observation and not a tested claim.
Anchoring framing-B windows on clock-kills against win-probability-matched non-decisive controls, neither point error nor coverage separates the clock-kill, and the contrast is not clean because treatment and control differ on realized |wpa|. The probability integral transform shows no boundary pile-up for any model: the per-model clock-kill PIT means are 0.46 to 0.49, with at most six percent of the mass above 0.9, indicating conservative rather than overconfident intervals. These are why the recovery result rests on the known answer and not on calibration.
References are listed below, with arXiv identifiers where no journal DOI is assigned.
- Extreme-Path Benchmarks for Sequential Probability Forecasts With Applications to Sports Win Probabilities (arXiv 2601.18774). Nearest prior art; ESPN NFL and NBA win-probability paths against a martingale benchmark, no foundation model, and an independent finding that NFL win probability is martingale-consistent.
- Kelly Betting as Bayesian Model Evaluation (arXiv 2602.09982). In-game win-probability model comparison.
- Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated? (arXiv 2510.16060). Calibration of foundation models in the coverage sense.
nflfastRandnfl_data_py, and the Open Source Football expected-points and win-probability methodology, for the data and the EP and WP definitions.- The foundation models used as ladder rungs: Chronos (Ansari et al., 2024), Chronos-2 (Amazon, 2025), TimesFM (Das et al., 2024), and TiRex (Auer et al., 2025, arXiv 2505.23719).
- Pedregosa et al. (2011), Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research 12, for the gradient-boosted and multilayer-perceptron ladder rungs.
- The structural-break literature contrasted in Section 2, which detects and dates breaks rather than forecasting through a known one: Bai and Perron (1998), Estimating and Testing Linear Models With Multiple Structural Changes, Econometrica 66(1); and Andrews (1993), Tests for Parameter Instability and Structural Change With Unknown Change Point, Econometrica 61(4).