Skip to content

Commit d41b05e

Browse files
hyperpolymathclaude
hyperpolymath
and
claude
authored
Enforce neural-boundary numeric provenance guardrail (P0) (#37)
**Makes the flagship guarantee real.** "No number is ever produced by the LLM" was, until now, prompt-only. This adds code-level enforcement plus the boundary hardening that guarantee depends on. ## What changed - **NEW `src/tools/guardrail.jl`** — `collect_numbers` (recursive harvest of every number a tool call produced, skipping Bool flags), `extract_numeric_tokens`, and `validate_numeric_provenance(text, tool_results, user_numbers; rtol)`. A numeric literal in the assistant's prose is legitimate iff it matches (within rtol, or after display-rounding) a harvested tool-result number, its ÷100/×100 percent variant, a user-supplied number, or is a small structural integer (0–12: df, counts, indices). Anything else is an **orphan** (a likely mollock). - **`chat.jl`** — records tool results per turn; runs the guardrail on the final assistant content; on orphans, retries **once** (restate using only tool-derived numbers) then appends a visible warning block. Also flags numeric answers given with **no** tool call. **Never rewrites the model's text — flags, never fabricates.** REPL turn body wrapped in try/catch so one bad turn can't kill the session. - **`lmstudio.jl` `process_tool_calls`** — per-tool-call try/catch → clean `role:"tool"` error payload (recover instead of crash); **forwards the previously-dropped `tools`** on the follow-up call; bounded multi-round loop (max 5) so compound queries actually chain. HTTP `connect_timeout=10, readtimeout=120, retry=false`. - **`executor.jl`** — clamps `n_reps`/`n_permutations` ≤ 100 000 and `k` ≤ 20; trailing `else` "Unknown type" guard on all 21 non-exhaustive inner sub-type dispatches (kills the silent-`null`→invented-number path); catch-all backtrace gated behind `STATISTIKLES_DEBUG`. - **NEW `test/guardrail_test.jl`** (wired into `runtests.jl`) — clean fixture passes; **injected-fabrication fixture is flagged** (`r=0.87`, `t=3.14159` orphaned while the real `42.73` is not); malformed-tool-call recovery (no HTTP); clamps; unknown-sub-type guards. ## Verification Full suite in WSL (Julia 1.10.11, serialized under flock): **4484 / 4484 passing** — 4404 baseline unchanged + 80 new guardrail tests. Zero regressions. ## Notes / deviations - The timeout-pattern reference file is `src/bridge/echidna_adapter.jl` (not `src/tools/`); exact kwargs from the spec were used regardless. - `k`-clamp applied to the three params literally named `k` (bayesian_em, ica, topic_modeling_nmf); `pca`'s `n_components` left unclamped (bounded by data dimensionality). - `else`-guards added to all non-exhaustive inner dispatches; already-exhaustive ones (chi_square, correlation, regression) left alone. Part of the production-readiness wave-1 remainder (W1-1, the P0). Implemented by an Opus agent, diff reviewed against the guardrail spec. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent b92dd09 commit d41b05e

7 files changed

Lines changed: 568 additions & 48 deletions

File tree

‎src/Statistikles.jl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ include("integrations/quantum_integration.jl") # QuantumCircuit.jl Bell tests
101101
include("tools/definitions.jl") # MCP / Function calling schemas
102102
include("tools/executor.jl") # Safe execution sandbox
103103
include("tools/lmstudio.jl") # Local LLM connectivity
104+
include("tools/guardrail.jl") # Neural-boundary numeric provenance enforcement
104105
include("tools/chat.jl") # Interactive session management
105106

106107
export main, run_examples, statistical_assistant_chat,

‎src/tools/chat.jl‎

Lines changed: 92 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -97,30 +97,108 @@ function statistical_assistant_chat()
9797
lowercase(input) == "help" && (print_help(); continue)
9898
lowercase(input) == "offline" && (run_examples(); continue)
9999

100-
push!(messages, Dict{String,Any}("role" => "user", "content" => input))
100+
# One malformed turn must never kill the session — isolate the turn body.
101+
try
102+
push!(messages, Dict{String,Any}("role" => "user", "content" => input))
101103

102-
print("\n Statistikles: ")
103-
response = call_lm_studio(messages, tools)
104+
print("\n Statistikles: ")
105+
response = call_lm_studio(messages, tools)
104106

105-
if haskey(response, "error")
106-
println("Error: $(response["error"])")
107-
pop!(messages)
108-
continue
109-
end
107+
if haskey(response, "error")
108+
println("Error: $(response["error"])")
109+
pop!(messages)
110+
continue
111+
end
110112

111-
final = process_tool_calls(response, messages)
112-
isnothing(final) && (final = response)
113+
pr = process_tool_calls(response, messages; tools=tools)
114+
final = pr.final
115+
116+
content = nothing
117+
if haskey(final, "choices") && !isempty(final["choices"])
118+
content = get(final["choices"][1]["message"], "content", nothing)
119+
end
113120

114-
if haskey(final, "choices") && !isempty(final["choices"])
115-
content = get(final["choices"][1]["message"], "content", nothing)
116121
if !isnothing(content)
122+
# NEURAL-BOUNDARY GUARDRAIL: every number in the reply must trace
123+
# back to a symbolic tool result before we trust (and print) it.
124+
user_numbers = extract_numeric_values(input)
125+
content = enforce_numeric_boundary!(
126+
content, messages, tools, pr.tool_results,
127+
user_numbers, pr.tool_calls_made)
117128
println(content)
118129
push!(messages, Dict{String,Any}("role" => "assistant", "content" => content))
119130
end
131+
catch e
132+
println("\n [turn error] $(sprint(showerror, e)) — continuing.")
133+
continue
120134
end
121135
end
122136
end
123137

138+
"""
139+
enforce_numeric_boundary!(content, messages, tools, tool_results,
140+
user_numbers, tool_calls_made) -> String
141+
142+
Run the numeric-provenance guardrail on an assistant reply. If unverified
143+
numbers remain, do ONE retry asking the model to restate using only numbers
144+
from the tool results (or, when no tool ran this turn, to compute them or
145+
refrain). If orphans still persist, return the reply with a clear warning block
146+
appended. The model's text is NEVER silently rewritten — orphans are flagged,
147+
never fabricated.
148+
"""
149+
function enforce_numeric_boundary!(content::AbstractString, messages, tools,
150+
tool_results, user_numbers, tool_calls_made::Bool)
151+
ok, orphans = validate_numeric_provenance(content, tool_results, user_numbers)
152+
ok && return String(content)
153+
154+
# ── Single retry ─────────────────────────────────────────────────────────
155+
retry_msg = if tool_calls_made
156+
"Your previous reply contained numeric value(s) that do not appear in any " *
157+
"tool result: " * join(orphans, ", ") * ". Restate your answer using ONLY " *
158+
"numbers returned by the tool calls above. Do not introduce any other " *
159+
"numeric value; if a needed number is missing, call the appropriate tool."
160+
else
161+
"Your previous reply asserted numeric value(s) (" * join(orphans, ", ") *
162+
") but NO symbolic computation was performed this turn, so none of them is " *
163+
"verified. Call the appropriate statistical tool to compute them, or restate " *
164+
"your answer without asserting specific numbers."
165+
end
166+
push!(messages, Dict{String,Any}("role" => "user", "content" => retry_msg))
167+
168+
new_content = String(content)
169+
retry_resp = call_lm_studio(messages, tools)
170+
if !haskey(retry_resp, "error")
171+
pr = process_tool_calls(retry_resp, messages; tools=tools)
172+
append!(tool_results, pr.tool_results)
173+
if haskey(pr.final, "choices") && !isempty(pr.final["choices"])
174+
c = get(pr.final["choices"][1]["message"], "content", nothing)
175+
isnothing(c) || (new_content = String(c))
176+
end
177+
end
178+
179+
ok2, orphans2 = validate_numeric_provenance(new_content, tool_results, user_numbers)
180+
ok2 && return new_content
181+
182+
return new_content * _guardrail_warning_block(orphans2, tool_calls_made)
183+
end
184+
185+
function _guardrail_warning_block(orphans, tool_calls_made::Bool)
186+
io = IOBuffer()
187+
println(io)
188+
println(io)
189+
println(io, " " * "!"^70)
190+
println(io, " UNVERIFIED NUMBERS — POSSIBLE MOLLOCK")
191+
tool_calls_made || println(io, " (no symbolic computation was performed this turn)")
192+
println(io, " The following number(s) in the reply above did not come from any")
193+
println(io, " symbolic tool result and could not be verified:")
194+
for o in orphans
195+
println(io, " - $o")
196+
end
197+
println(io, " Trust only numbers produced by [symbolic] tool calls.")
198+
print(io, " " * "!"^70)
199+
return String(take!(io))
200+
end
201+
124202
function print_help()
125203
println("""
126204
@@ -201,7 +279,8 @@ function main()
201279

202280
try
203281
test_response = HTTP.request("GET", "$BASE_URL/models", HEADERS;
204-
status_exception=false)
282+
status_exception=false,
283+
connect_timeout=10, readtimeout=120, retry=false)
205284
if test_response.status == 200
206285
println("\n Connected to LM Studio ($MODEL)")
207286
else

‎src/tools/executor.jl‎

Lines changed: 60 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ function execute_tool(tool_name::String, arguments::Dict)
2626
return t_test_paired(g1, convert(Vector{Float64}, arguments["group2"]); alpha)
2727
elseif tt == "one_sample"
2828
return t_test_one_sample(g1, Float64(get(arguments, "mu0", 0.0)); alpha)
29+
else
30+
return Dict{String,Any}("error" => "Unknown type '$tt' for t_test")
2931
end
3032

3133
elseif tool_name == "anova"
@@ -100,6 +102,7 @@ function execute_tool(tool_name::String, arguments::Dict)
100102
stat_name = get(arguments, "statistic", "mean")
101103
stat_fn = stat_name == "mean" ? mean : stat_name == "median" ? median : var
102104
reps = Int(get(arguments, "n_reps", 1000))
105+
reps > 100_000 && return Dict{String,Any}("error" => "n_reps=$reps exceeds maximum 100000")
103106
return bootstrap_ci(data, stat_fn; n_reps=reps)
104107

105108
elseif tool_name == "time_series"
@@ -113,6 +116,8 @@ function execute_tool(tool_name::String, arguments::Dict)
113116
return Dict("acf" => autocorrelation(data, lag))
114117
elseif arguments["type"] == "dtw"
115118
return Dict("dtw_distance" => dynamic_time_warping(data, convert(Vector{Float64}, arguments["target"])))
119+
else
120+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for time_series")
116121
end
117122

118123
elseif tool_name == "information_theory"
@@ -122,6 +127,8 @@ function execute_tool(tool_name::String, arguments::Dict)
122127
p = convert(Vector{Float64}, arguments["p"])
123128
q = convert(Vector{Float64}, arguments["q"])
124129
return Dict("kl_divergence" => kl_divergence(p, q))
130+
else
131+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for information_theory")
125132
end
126133

127134
elseif tool_name == "survival_analysis"
@@ -132,6 +139,8 @@ function execute_tool(tool_name::String, arguments::Dict)
132139
return log_rank_test(convert(Vector{Float64}, arguments["times"]),
133140
convert(Vector{Bool}, arguments["events"]),
134141
arguments["groups"])
142+
else
143+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for survival_analysis")
135144
end
136145

137146
elseif tool_name == "meta_analysis"
@@ -147,6 +156,8 @@ function execute_tool(tool_name::String, arguments::Dict)
147156
return Dict("distances" => mahalanobis_distance(X))
148157
elseif arguments["type"] == "huber"
149158
return Dict("estimate" => huber_m_estimator(convert(Vector{Float64}, arguments["data"])))
159+
else
160+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for robust_stats")
150161
end
151162

152163
elseif tool_name == "causal_inference"
@@ -162,6 +173,8 @@ function execute_tool(tool_name::String, arguments::Dict)
162173
return regression_discontinuity(convert(Vector{Float64}, arguments["y"]),
163174
convert(Vector{Float64}, arguments["x"]),
164175
Float64(arguments["threshold"]))
176+
else
177+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for causal_inference")
165178
end
166179

167180
elseif tool_name == "spatial_stats"
@@ -170,6 +183,8 @@ function execute_tool(tool_name::String, arguments::Dict)
170183
raw_w = arguments["w"]
171184
W = Matrix{Float64}(hcat([convert(Vector{Float64}, col) for col in raw_w]...))
172185
return morans_i(x, W)
186+
else
187+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for spatial_stats")
173188
end
174189

175190
elseif tool_name == "machine_learning"
@@ -180,6 +195,8 @@ function execute_tool(tool_name::String, arguments::Dict)
180195
raw_x = arguments["x"]
181196
X = Matrix{Float64}(hcat([convert(Vector{Float64}, col) for col in raw_x]...))
182197
return random_forest_proxy(X, convert(Vector{Float64}, arguments["y"]))
198+
else
199+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for machine_learning")
183200
end
184201

185202
elseif tool_name == "nlp_symbolic"
@@ -189,7 +206,11 @@ function execute_tool(tool_name::String, arguments::Dict)
189206
elseif arguments["type"] == "topic_modeling"
190207
raw_x = arguments["x"]
191208
X = Matrix{Float64}(hcat([convert(Vector{Float64}, col) for col in raw_x]...))
192-
return topic_modeling_nmf(X; k=Int(get(arguments, "k", 3)))
209+
k = Int(get(arguments, "k", 3))
210+
k > 20 && return Dict{String,Any}("error" => "k=$k exceeds maximum component count 20")
211+
return topic_modeling_nmf(X; k=k)
212+
else
213+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for nlp_symbolic")
193214
end
194215

195216
elseif tool_name == "advanced_modeling"
@@ -199,16 +220,24 @@ function execute_tool(tool_name::String, arguments::Dict)
199220
elseif arguments["type"] == "ordinal_logistic"
200221
X = Matrix{Float64}(hcat([convert(Vector{Float64}, col) for col in arguments["x"]]...))
201222
return ordinal_logistic_regression(X, convert(Vector{Int}, arguments["y"]))
223+
else
224+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for advanced_modeling")
202225
end
203226

204227
elseif tool_name == "signal_processing"
205228
if arguments["type"] == "ica"
206229
X = Matrix{Float64}(hcat([convert(Vector{Float64}, col) for col in arguments["x"]]...))
207-
return independent_component_analysis(X; k=Int(get(arguments, "k", 2)))
230+
k = Int(get(arguments, "k", 2))
231+
k > 20 && return Dict{String,Any}("error" => "k=$k exceeds maximum component count 20")
232+
return independent_component_analysis(X; k=k)
233+
else
234+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for signal_processing")
208235
end
209236

210237
elseif tool_name == "bayesian_em"
211-
return expectation_maximization_normal(convert(Vector{Float64}, arguments["data"]), Int(arguments["k"]))
238+
k = Int(arguments["k"])
239+
k > 20 && return Dict{String,Any}("error" => "k=$k exceeds maximum component count 20")
240+
return expectation_maximization_normal(convert(Vector{Float64}, arguments["data"]), k)
212241

213242
elseif tool_name == "functional_data"
214243
X = Matrix{Float64}(hcat([convert(Vector{Float64}, col) for col in arguments["x"]]...))
@@ -219,6 +248,8 @@ function execute_tool(tool_name::String, arguments::Dict)
219248
return mcnemar_test(Int(arguments["b"]), Int(arguments["c"]))
220249
elseif arguments["type"] == "padic"
221250
return Dict("valuation" => padic_valuation(Int(arguments["n"]), Int(arguments["p"])))
251+
else
252+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for algebraic_stats")
222253
end
223254

224255
elseif tool_name == "representation_stats"
@@ -228,6 +259,8 @@ function execute_tool(tool_name::String, arguments::Dict)
228259
a = Tuple{Float64, Float64}(arguments["interval_a"])
229260
b = Tuple{Float64, Float64}(arguments["interval_b"])
230261
return interval_overlap_test(a, b)
262+
else
263+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for representation_stats")
231264
end
232265

233266
elseif tool_name == "non_classical_prob"
@@ -237,6 +270,8 @@ function execute_tool(tool_name::String, arguments::Dict)
237270
return Dict("result" => tropical_dot_product(v1, v2))
238271
elseif arguments["type"] == "bell_test"
239272
return Dict("chsh_s" => bell_test_chsh(convert(Vector{Float64}, arguments["correlations"])))
273+
else
274+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for non_classical_prob")
240275
end
241276

242277
elseif tool_name == "structured_dynamic"
@@ -250,12 +285,16 @@ function execute_tool(tool_name::String, arguments::Dict)
250285
return Dict("dimension" => box_counting_dimension(img))
251286
elseif arguments["type"] == "hurst"
252287
return Dict("hurst" => hurst_exponent(convert(Vector{Float64}, arguments["data"])))
288+
else
289+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for structured_dynamic")
253290
end
254291

255292
elseif tool_name == "unconventional_frameworks"
256293
if arguments["type"] == "rough_set"
257294
return rough_set_approximations(convert(Vector{Int}, arguments["features"]),
258295
convert(Vector{Int}, arguments["target"]))
296+
else
297+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for unconventional_frameworks")
259298
end
260299

261300
elseif tool_name == "pre_suite"
@@ -282,13 +321,16 @@ function execute_tool(tool_name::String, arguments::Dict)
282321
raw_d = arguments["data"]
283322
d = Matrix{Int}(hcat([convert(Vector{Int}, col) for col in raw_d]...))
284323
return cochrans_q(d)
324+
else
325+
return Dict{String,Any}("error" => "Unknown type '$tt' for nonparametric_test")
285326
end
286327

287328
elseif tool_name == "permanova"
288329
raw_dm = arguments["distance_matrix"]
289330
dm = Matrix{Float64}(hcat([convert(Vector{Float64}, row) for row in raw_dm]...)')
290331
labels = arguments["group_labels"]
291332
n_perm = Int(get(arguments, "n_permutations", 999))
333+
n_perm > 100_000 && return Dict{String,Any}("error" => "n_permutations=$n_perm exceeds maximum 100000")
292334
alpha = Float64(get(arguments, "alpha", 0.05))
293335
return permanova(dm, labels; n_permutations=n_perm, alpha=alpha)
294336

@@ -324,6 +366,8 @@ function execute_tool(tool_name::String, arguments::Dict)
324366
return test_normality(convert(Vector{Float64}, arguments["data"]))
325367
elseif arguments["type"] == "levene"
326368
return levenes_test([convert(Vector{Float64}, g) for g in arguments["groups"]])
369+
else
370+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for test_assumptions")
327371
end
328372

329373
elseif tool_name == "bayesian_analysis"
@@ -390,6 +434,8 @@ function execute_tool(tool_name::String, arguments::Dict)
390434
raw = arguments["items"]
391435
items = Matrix{Float64}(hcat([convert(Vector{Float64}, row) for row in raw]...)')
392436
return item_analysis(items, vec(sum(items, dims=2)))
437+
else
438+
return Dict{String,Any}("error" => "Unknown type '$mt' for measurement_analysis")
393439
end
394440

395441
elseif tool_name == "validity_assessment"
@@ -413,6 +459,8 @@ function execute_tool(tool_name::String, arguments::Dict)
413459
raw = arguments["ratings_matrix"]
414460
mat = Matrix{Float64}(hcat([convert(Vector{Float64}, row) for row in raw]...)')
415461
return intraclass_correlation(mat; icc_type=get(arguments, "icc_type", "ICC(2,1)"))
462+
else
463+
return Dict{String,Any}("error" => "Unknown type '$irr' for inter_rater_reliability")
416464
end
417465

418466
elseif tool_name == "qualitative_analysis"
@@ -435,15 +483,20 @@ function execute_tool(tool_name::String, arguments::Dict)
435483
proportion=Float64(get(arguments, "proportion", 0.5)),
436484
confidence=Float64(get(arguments, "confidence", 0.95)),
437485
population=pop)
486+
else
487+
return Dict{String,Any}("error" => "Unknown type '$(arguments["type"])' for sampling_design")
438488
end
439489

440490
else
441491
return Dict{String,Any}("error" => "Unknown tool: $tool_name")
442492
end
443493
catch e
444-
return Dict{String,Any}(
445-
"error" => "Tool execution failed: $(string(e))",
446-
"trace" => sprint(showerror, e, catch_backtrace())
447-
)
494+
# Stable, concise error string. The full backtrace is only attached when
495+
# STATISTIKLES_DEBUG is set, so normal output stays clean and stable.
496+
err = Dict{String,Any}("error" => "Tool execution failed: $(string(e))")
497+
if get(ENV, "STATISTIKLES_DEBUG", "") != ""
498+
err["trace"] = sprint(showerror, e, catch_backtrace())
499+
end
500+
return err
448501
end
449502
end

0 commit comments

Comments
 (0)