Skip to content

Improve prompt validation with new static warnings for inputs, cache, provider, and tools#11

Merged
PredictabilityAtScale merged 1 commit into
mainfrom
codex/review-prompt-templates-for-validation-issues
Apr 24, 2026
Merged

Improve prompt validation with new static warnings for inputs, cache, provider, and tools#11
PredictabilityAtScale merged 1 commit into
mainfrom
codex/review-prompt-templates-for-validation-issues

Conversation

@PredictabilityAtScale
Copy link
Copy Markdown
Owner

Motivation

  • Improve prompt static validation to surface risky prompt definitions and offer actionable guidance for input sizing, hardening, provider cache/model hygiene, and inline tool metadata.
  • Make unknown front-matter suggestions more comprehensive by adding provider_options to known keys and update docs to enumerate the new warning codes.

Description

  • Add POK046 to warn when a template uses variables but context.inputs is not declared and suggest declaring context.inputs for policy validation via the validateAsset check.
  • Add POK040 and POK041 to flag context inputs that look unbounded (common names without max_size) and inputs missing hardening validators, using a RISKY_UNBOUNDED_INPUT_NAMES heuristic.
  • Add provider/cache/model hygiene checks emitting POK042 (missing provider cache config), POK043 (conflicting Gemini/Google cached_content), POK044 (provider configured without model), and POK045 (environment/tier cache overrides missing).
  • Add inline tool quality checks that emit POK047 when description or input_schema are missing for inline tool definitions.
  • Update docs/validation.md to document all new codes and add provider_options to known front-matter keys.
  • Update unit tests in tests/validation.test.ts to cover the new warnings and behaviors.

Testing

  • Ran unit tests in tests/validation.test.ts covering POK046, POK040/POK041, POK042/POK044, POK043, and POK047 scenarios.
  • All updated validation tests passed locally when running npm test.
  • Existing schema and regex compilation tests continue to pass after the changes.

Codex Task

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f41130070

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}

for (const [envName, overrides] of Object.entries(asset.environments ?? {})) {
if (asset.cache && !overrides.cache) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard override entries before reading cache

validateAsset now dereferences overrides.cache without checking that each environments entry is an object. If a caller passes malformed input (for example { environments: { prod: null } }), safeParse correctly identifies schema issues, but this line throws a TypeError before validation results are returned. That turns a recoverable validation failure into a hard crash; the same pattern appears in the tier loop as well.

Useful? React with 👍 / 👎.


for (const tool of asset.tools ?? []) {
if (typeof tool !== 'string') {
if (!tool.description) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard inline tool checks against non-object values

The new inline-tool warning path assumes every non-string tool entry is an object and directly reads tool.description. For malformed assets like { tools: [null] }, validation now throws (Cannot read properties of null) instead of returning POK001 schema errors. Since this function already calls safeParse, it should tolerate bad shapes and keep reporting structured validation output.

Useful? React with 👍 / 👎.

@PredictabilityAtScale PredictabilityAtScale merged commit 94d98cf into main Apr 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant