Feature - new apis

[drorIvry]

Description of change

Pull-Request Checklist

• Code is up-to-date with the main branch
• npm run lint passes with this change
• npm run test passes with this change
• This pull request links relevant issues as Fixes #0000
• There are new or updated unit tests validating the change
• Documentation has been updated to reflect this change
• The new commits follow conventions outlined in the conventional commit spec

Summary by CodeRabbit

• New Features

  • Publicly export an LLMMessage type.
  • Allow optional messages and availableTools when invoking evaluations.
  • Add new evaluation configuration modes: assertions, hallucinations, grounding, multi-turn policy, policy target, and allowed topics.

• Bug Fixes / Improvements

  • Evaluation API endpoints and telemetry/tracing targets updated to the new service base URLs.

• Chores

  • Package version bumped to 1.7.0.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Bumps package version to 1.7.0, changes default API host to api.qualifire.ai, updates telemetry/tracing base URL, expands evaluation/invoke payloads and endpoints to /api/v1/..., re-exports LLMMessage, and extends public invokeEvaluation signature to accept optional messages and availableTools.

Changes

Cohort / File(s)	Summary
Version Update package.json	Version bumped from 1.6.0 to 1.7.0.
Evaluation API & Public API src/index.ts	Default host changed to https://api.qualifire.ai; telemetry/tracing base URL adjusted. Endpoints moved to /api/v1/.... Evaluation payloads extended with messages, available_tools, assertions_mode, hallucinations_mode, grounding_mode, grounding_multi_turn_mode, policy_multi_turn_mode, policy_target/policyTarget, allowed_topics/allowedTopics. invokeEvaluation signature now accepts optional messages and availableTools. LLMMessage re-exported publicly.

Sequence Diagram(s)

sequenceDiagram
  participant SDK as Client SDK
  participant Conv as Converter
  participant API as Qualifire API
  participant Eval as Evaluation Service

  SDK->>Conv: prepare/convert request (messages, tools, modes...)
  alt converter path
    Conv->>API: POST /api/v1/evaluation/evaluate (converted payload)
  else direct/backward-compatible path
    SDK->>API: POST /api/v1/evaluation/evaluate (messages, available_tools, modes...)
  end
  API->>Eval: enqueue/evaluate request
  Eval-->>API: evaluation result
  API-->>SDK: return EvaluationResponse

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• FIRE-571 FIRE-572 FIRE-589 | Tracing + TSQ Detector #19 — adds messages and available_tools support and adjusts evaluation/invoke flows.
• feat: metadata #27 — extends invoke/evaluation payloads and propagates metadata/allowedTopics.
• Hotfix | New evaluations-api schema fix #21 — modifies evaluation API surface and public typings; includes version bump.

Suggested reviewers

• yuval-qf

Poem

🐰 I hopped a version up to one-point-seven,
Messages and modes now dance like levin.
Endpoints shifted, types set free,
Tools and traces join the spree.
🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The title 'Feature - new apis' is vague and generic, using non-descriptive terms that don't convey meaningful information about the specific changes made.	Replace with a more specific title that reflects the main change, such as 'Update Qualifire API endpoints and add invoke evaluation support' or 'Upgrade to v1 API endpoints and add message-based evaluation'.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/evaluate-invoke-FIRE-1110

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

src/index.ts (2)

45-58: ⚠️ Potential issue | 🟠 Major

Default baseUrl should remain the proxy endpoint.
Switching the default to the direct API changes expected routing; keep the proxy as the default and allow the direct API via override instead.

🔧 Suggested fix

-    const qualifireBaseUrl =
-      baseUrl || process.env.QUALIFIRE_BASE_URL || 'https://api.qualifire.ai';
+    const qualifireBaseUrl =
+      baseUrl || process.env.QUALIFIRE_BASE_URL || 'https://proxy.qualifire.ai';

As per coding guidelines: Qualifire API endpoint should be {baseUrl}/api/evaluation/evaluate with default baseUrl of https://proxy.qualifire.ai.

---

415-441: ⚠️ Potential issue | 🟡 Minor

Guard against empty invokeEvaluation payloads.
With input/output optional, it’s now possible to call this with no model data and no messages. Add a quick validation to avoid sending invalid requests.

🛡️ Suggested fix

   }): Promise<EvaluationResponse | undefined> => {
+    if (!input && !output && (!messages || messages.length === 0)) {
+      throw new Error('invokeEvaluation requires input/output or messages.');
+    }
     const url = `${this.baseUrl}/api/evaluation/invoke`;

🤖 Fix all issues with AI agents

In `@src/index.ts`:
- Around line 268-277: The converter-based path for EvaluationRequestV2 is
missing propagation of new policy fields (policy_multi_turn_mode, policy_target,
allowed_topics and the other policy/grounding/assertion mode fields) from
evaluationProxyAPIRequest; update the converter payload construction used by
EvaluationRequestV2 to include the same mappings as the backward-compat path
(e.g., map evaluationProxyAPIRequest.policyMultiTurnMode ->
policy_multi_turn_mode, policyTarget -> policy_target, allowedTopics ->
allowed_topics, and likewise for assertions_mode, hallucinations_mode,
grounding_mode, grounding_multi_turn_mode) so converter callers send the new
fields as well.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Propagate new policy mode fields to converter-based requests.
These fields are only sent in the backward-compat path, so EvaluationRequestV2 callers using converters won’t send them. Consider adding them to the converter payload as well.

➕ Suggested fix

       tuq_mode: EvaluationRequestV2.tuqMode ?? EvaluationRequestV2.tsqMode,
       assertions: EvaluationRequestV2.assertions,
+      assertions_mode: EvaluationRequestV2.assertionsMode,
+      hallucinations_mode: EvaluationRequestV2.hallucinationsMode,
+      grounding_mode: EvaluationRequestV2.groundingMode,
+      grounding_multi_turn_mode: EvaluationRequestV2.groundingMultiTurnMode,
+      policy_multi_turn_mode: EvaluationRequestV2.policyMultiTurnMode,
+      policy_target: EvaluationRequestV2.policyTarget,
       topic_scoping_mode: EvaluationRequestV2.topicScopingMode,

🤖 Prompt for AI Agents

In `@src/index.ts` around lines 268 - 277, The converter-based path for
EvaluationRequestV2 is missing propagation of new policy fields
(policy_multi_turn_mode, policy_target, allowed_topics and the other
policy/grounding/assertion mode fields) from evaluationProxyAPIRequest; update
the converter payload construction used by EvaluationRequestV2 to include the
same mappings as the backward-compat path (e.g., map
evaluationProxyAPIRequest.policyMultiTurnMode -> policy_multi_turn_mode,
policyTarget -> policy_target, allowedTopics -> allowed_topics, and likewise for
assertions_mode, hallucinations_mode, grounding_mode, grounding_multi_turn_mode)
so converter callers send the new fields as well.