fix: stop Vakil Friend citing law when the RAG service is unavailable - #1664
Open
Anijesh wants to merge 1 commit into
Open
fix: stop Vakil Friend citing law when the RAG service is unavailable#1664Anijesh wants to merge 1 commit into
Anijesh wants to merge 1 commit into
Conversation
When lawgpt-service was down, RagService returned the same "No specific legal context found." string it returns for a genuine no-match, so VakilFriendService could not tell "the corpus has nothing on this" from "nothing answered". Either way the system prompt was left untouched and Llama answered from memory, confidently producing IPC/BNS section numbers and case citations that do not exist. RagService: - Report the outcome of a retrieval, not just its text, via a RagContext record carrying GROUNDED / NO_MATCH / UNAVAILABLE - Treat a non-200 response as UNAVAILABLE rather than silently as no-match VakilFriendService: - Default the turn to UNAVAILABLE, so a missing RagService bean and an unreachable LawGPT both count as ungrounded instead of authoritative - Route chat-title and JSON-extraction completions through a separate callGroqAPIForUtility(), so the new guardrail cannot prefix a title with a disclaimer or break the strict-JSON contract - Stop logging the raw user query alongside the retrieval result LegalGroundingGuard (new): - Input side: appends either the retrieved law or an explicit "you have no verified sources" instruction to the system prompt, telling the model to open with "I'm unable to retrieve verified legal references right now.", withhold every section number and citation, and point the user at India Code, eCourts or an advocate - Output side: a prompt is guidance, not a guarantee, so ungrounded responses are scanned for statutory and law-report citations. Any found are redacted and the reply is wrapped in the unverified notice - Ordinary case-filing turns that collect names, dates and evidence pass through untouched, and the "### CASE SUMMARY START ###" block survives redaction so filing still works Tests: - 27 cases for citation detection, redaction and prompt selection - 6 cases for the retrieval statuses, including an unreachable LawGPT Closes viru0909-dev#1622
|
Someone is attempting to deploy a commit to the CodeBlooded's projects Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
Hi @Anijesh, thanks for contributing to Nyay Setu! 🎉 I have automatically:
Our workflows will now analyze your changes to classify:
Tip Ensure your PR description references the issue it resolves (e.g. Happy coding! 🚀 |
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
RagService.findRelevantContext()returned the string"No specific legal context found."in two very different situations: LawGPT answered and genuinely had nothing, and LawGPT never answered at all.VakilFriendServicecould not tell them apart, so in both cases it leftSYSTEM_PROMPTuntouched and let Llama answer from memory — which is where the fabricated IPC/BNS sections and case citations come from.This PR makes the retrieval outcome explicit and guards the answer in both directions.
Issue Resolved
Closes #1622
Changes Made
RagService— say why there is no contextA new
RagContextrecord carries the outcome alongside the text:GROUNDEDNO_MATCHUNAVAILABLEA non-200 response now counts as
UNAVAILABLEinstead of being silently folded into "no match".searchPrecedentsis unchanged.VakilFriendService— treat "no verified corpus" as the defaultThe turn starts at
UNAVAILABLE, so a missingRagServicebean (rag.enabled=false) and a dead LawGPT are both handled as ungrounded rather than authoritative. Previously therag.enabled=falsedeployment hit exactly the same hallucination path and nobody noticed.LegalGroundingGuard(new) — guard both directionsInput side. When the turn is not grounded, the system prompt gains an explicit instruction — the fix the issue suggested. It tells the model to open with "I'm unable to retrieve verified legal references right now.", never state or guess a section number, act number, case name or law-report citation, never state a punishment or limitation period, and close by pointing the user at India Code, eCourts or a qualified advocate.
It is deliberately scoped to legal questions. Most turns of the case-filing flow just collect names and dates, and burying those under a disclaimer would be worse UX than the bug.
Output side. A system prompt is guidance, not a guarantee — models still emit confident citations after being told not to. Ungrounded responses are therefore scanned for statutory references (
Section 302,sections 302, 420 and 34,u/s 420,Article 21,IPC 302,BNSS 173) and law-report citations (AIR 1973 SC 1461,(2017) 10 SCC 1,2019 SCC OnLine SC 1234). If any are found, they are redacted and the reply is wrapped in the unverified notice plus the "consult an official source" advice.Citation-free responses are returned byte-for-byte unchanged, and the
### CASE SUMMARY START ###block survives redaction so case filing keeps working.Two details worth a reviewer's attention
1.
callGroqAPIhad three callers, not one. Besides the chat path,generateSessionTitle()andgenerateCaseTitle()call it with an empty context. Attaching the guardrail to "no context" alone would have made the model prefix chat titles with a disclaimer and break the strict-JSON contractgenerateCaseTitleparses. Those two now go throughcallGroqAPIForUtility(), which applies neither the context block nor the guardrail.2. Redaction is deliberately slightly over-broad. If the assistant echoes a citation the user typed ("the police booked me under section 420"), that gets redacted too. That is the intended trade-off — the platform should never restate an unverified provision, even the user's own.
How to Test
lawgpt-service.RAG retrieval status for this turn: UNAVAILABLE.[reference removed - could not be verified]and a⚠️warning is logged.lawgpt-serviceand repeat — the answer is grounded again and nothing is redacted.Testing Completed
mvn test -Dtest='LegalGroundingGuardTest,RagServiceTest'— 33 passedmain: 27 errors before, 27 errors after — identical set, all@SpringBootTestcontext-load failures needing a database. No regressions from this PR.Note:
mvn testdoes not currently succeed onmainThree pre-existing problems, none related to this issue, that I had to patch locally to run anything:
pom.xmlis missingcom.networknt:json-schema-validator.GroqResponseValidatorandVakilFriendGroqValidatorService(added in feat(ai): validated LLM output for Vakil-Friend case filing #1514) importcom.networknt.schema, somvn compilefails outright. The one-line fix:CaseServiceTest.javareferencesLegalCase/LegalCaseRepository, which no longer exist.SecurityIntegrationTest.javaimportsorg.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc, which moved in Spring Boot 4.I kept all three out of this PR to keep it scoped to #1622. Happy to add the dependency here or open a separate PR for all three — just say which you prefer.
Breaking Changes
None to any HTTP API.
RagService.findRelevantContext(String, int)is replaced byretrieveContext(String, int)returningRagContext; it had exactly one caller in the codebase, which is updated in this PR.