RAG retrieval gap: statute-boundary chunking causes ORS 90.325 / ORS 90.425 to be unreliably retrieved

[yangm2]

Problem

Two Oregon statutes that the agent needs to answer correctly — ORS 90.325 (tenant damage liability) and ORS 90.425 (abandoned personal property) — exist verbatim in our clean corpus but are not reliably surfaced by the Vertex AI Search retriever when the agent issues natural-language queries about them.

This was measured empirically by re-issuing the agent's real eval queries at production retrieval params (3 docs × 3 extractive segments) and checking whether the verbatim statutory text appeared in any returned segment:

Statute	Topic	Hit rate (production params)
ORS 90.325(3)(b)	DV victim not liable for perpetrator damage	0 / 4 queries
ORS 90.425(6)(b)	Abandoned-property contact window	2 / 4 queries (coin-flip)
ORS 90.394(1)	Week-to-week nonpayment fifth-day rule	3 / 3 ✓
PCC 30.01.087	Portland security deposit interest	2 / 2 ✓

The failing statutes are not missing from the corpus — they appear verbatim in backend/scripts/documents/or/2025/ORS090.txt. The problem is at the indexing / chunking layer.

Why it happens

ORS 90.txt is a large flat file. When Vertex AI Search indexes it, it cuts the text into chunks at boundaries it infers from the raw text — it does not know that "ORS 90.325(3)(b)" is a self-contained legal unit. As a result:

• ORS 90.325(3)(b) sits in a section surrounded by general tenant-damage and habitability provisions. When a user describes a domestic violence situation, the agent queries something like "tenant liability for damages caused by domestic violence perpetrator". The retriever fetches ORS 90.453–90.459 (the DV protection statutes, which are dense with DV terminology) instead of ORS 90.325, because 90.325's surrounding context doesn't carry the same DV signal density. Hit rate: 0/4.

• ORS 90.425(6)(b) ("not less than five days after personal delivery or eight days after mailing") lives inside a long section covering all aspects of abandoned personal property. The five/eight-day contact window lands in some chunks but not others depending on where the chunk boundary fell. Hit rate: 2/4.

The contrast is informative: ORS 90.394(1) (3/3) and PCC 30.01.087 (2/2) retrieve cleanly because those sections are shorter and more self-contained. The retriever consistently captures them because the relevant text isn't split across a chunk boundary.

---

Affected scenarios and magnitude

The eval dataset (tenant-legal-qa-scenarios) includes two scenarios directly affected. Legal correctness is scored 0.0 / 0.5 / 1.0 per run; mean is across 10 runs.

S3 — DV victim damage liability

"My ex attacked me and in the process destroyed several cabinets and a door. My landlord is now saying I'm responsible for the damage and could be evicted. Am I really responsible?"

The correct answer requires ORS 90.325(3)(b): a tenant is not responsible for damage caused by a DV perpetrator. This text retrieves 0/4 times.

Eval history:

Experiment	S3 mean	Notes
baseline-f7309242 (April 2026)	0.92	STOPGAP present
extractive-segments-dfe32555	0.95	STOPGAP present
extractive-segments-plus-clarifications-f3145c21	1.00	STOPGAP present
fewer-STOPGAPs-d5fd1f2a (current)	0.95	STOPGAP present

S3 only scores at all because the STOPGAP supplies the statutory text the retriever cannot. Without the STOPGAP, the agent would either answer from general tenant-damage liability principles (harmful — it would tell a DV victim they owe damages) or hedge with no clear answer. This is one of the highest-stakes failures in the eval suite: a wrong answer could lead a domestic violence survivor to believe they owe money to their landlord for damage caused by their abuser.

S1 — Abandoned personal property timing

"If I've moved out but there are some things I forgot, can my landlord just throw them away?"

The correct answer requires ORS 90.425(6)(b): the tenant must contact the landlord within a specific window (not less than 5 days after personal delivery, or 8 days after mailing of the notice). This text retrieves 2/4 times — making answers inconsistent across runs.

Eval history:

Experiment	S1 mean	Notes
baseline-f7309242 (April 2026)	0.88	STOPGAP present
extractive-segments-dfe32555	0.65	STOPGAP present, segment-mode introduced — some disruption
extractive-segments-plus-clarifications-f3145c21	0.80	STOPGAP present
fewer-STOPGAPs-d5fd1f2a (current)	0.95	STOPGAP present

The coin-flip retrieval rate (2/4) shows up as high run-to-run variance. Without the STOPGAP, about half of S1 runs would omit or misstate the contact window — which matters because giving a tenant the wrong deadline could cause them to lose access to their property.

---

Current workaround

backend/tenantfirstaid/system_prompt.md contains STOPGAP blocks — verbatim quotes from the corpus pasted directly into the system prompt so the model has the statutory text regardless of retrieval:

tenantfirstaid/backend/tenantfirstaid/system_prompt.md

Lines 36 to 37 in 66948b0

- **STOPGAP — [ORS 90.325](https://oregon.public.law/statutes/ors_90.325)(3)(b) victim damage liability.** "A tenant is not responsible for damage that results from... conduct by a perpetrator relating to domestic violence, sexual assault, bias crime or stalking."

- [ORS 90.325](https://oregon.public.law/statutes/ors_90.325)(4): "For damage that results from conduct by a perpetrator relating to domestic violence, sexual assault, bias crime or stalking, a landlord may require a tenant to provide verification that the tenant or a member of the tenant's household is a victim of domestic violence, sexual assault, bias crime or stalking as provided by ORS 90.453."

tenantfirstaid/backend/tenantfirstaid/system_prompt.md

Lines 44 to 47 in 66948b0

	- **STOPGAP — [ORS 90.425](https://oregon.public.law/statutes/ors_90.425)(3),(6)(b),(8) abandoned personal property notice and retrieval windows.**
	- Service rules: [ORS 90.155](https://oregon.public.law/statutes/ors_90.155)(1): "Except as provided in ORS 90.300, 90.315, 90.425 and 90.675, where this chapter requires written notice, service or delivery of that written notice shall be executed by one or more of the following methods." [ORS 90.425](https://oregon.public.law/statutes/ors_90.425)(3): "the landlord must give a written notice to the tenant that must be: (a) Personally delivered to the tenant; or (b) Sent by first class mail addressed and mailed to the tenant."
	- Contact window [ORS 90.425](https://oregon.public.law/statutes/ors_90.425)(6)(b): the tenant must contact the landlord "not less than five days after personal delivery or eight days after mailing of the notice."
	- Removal window [ORS 90.425](https://oregon.public.law/statutes/ors_90.425)(8): after the tenant responds, "the landlord must make that personal property available for removal... by appointment at reasonable times during the 15 days... following the date of the response."

This workaround has real costs:

• Token overhead and attention dilution: STOPGAP blocks consume prompt tokens. A longer system prompt causes "lost-in-the-middle" degradation — material in the middle of a long prompt receives less model attention. We measured this as a contributing factor in score regressions on other scenarios (S0, S1) when the prompt grew in earlier experiments.
• Maintenance burden: Every time the corpus is updated for a new ORS edition, the STOPGAP quotes need to be manually kept in sync with the source text.
• Scalability: We can't STOPGAP every statute that might have a chunking gap. As coverage expands to more legal topics and jurisdictions, this approach doesn't scale.

Two other STOPGAP blocks (ORS 90.394 and PCC 30.01.087) were already removed after evaluations confirmed those statutes retrieve reliably. The two remaining STOPGAPs (90.325 and 90.425) cannot safely be removed until this underlying issue is fixed.

---

Possible solutions

Option A: Section-aware pre-chunking (split before indexing)

Pre-process the ORS .txt files before uploading to GCS, splitting them so each file (or clearly delimited chunk) contains exactly one ORS section — its catchline, all subsections, and any annotations — rather than uploading the full ORS090.txt monolith.

How: Write a preprocessing script that parses the ORS text by section headings (e.g. 90.325 Liability of tenant for damage.) and emits one file per section, named ORS90.325.txt. The existing backend/scripts/ ingestion pipeline uploads these to GCS; Vertex AI Search indexes each file as its own document, guaranteeing the section's operative text lands in a single indexable unit.

Tradeoffs:

• ✅ Addresses the root cause directly — retrieval quality improves for all ORS sections, not just the two with STOPGAPs
• ✅ Compatible with the existing ingestion pipeline with minimal changes
• ✅ Allows the two remaining STOPGAPs to be removed from the system prompt
• ⚠️ Cross-section references ("as defined in ORS 90.100") lose their neighboring context at retrieval time — may reduce quality for questions that span multiple sections
• ⚠️ Requires a reliable section-boundary parser for the ORS text format (section headings aren't perfectly consistent across ORS chapters)
• ⚠️ Requires a full datastore rebuild and re-evaluation after the change
• Related: see Revision-controlled corpus-cfg.TOML for RAG deployment + chunking/granularity flags #354 for chunking/granularity config work

Option B: Structured corpus format (JSON/JSONL with section metadata)

Convert the flat .txt files to a structured format (JSONL) where each record is one ORS section with fields like section, title, text, and jurisdiction. Upload these as structured documents to Vertex AI Search, which can use the metadata fields for filtering and boost the section heading as a retrieval signal.

How: Write a converter that parses ORS090.txt into JSONL records. Update the Vertex AI Search datastore schema to index the section and title fields. This unlocks schema-level retrieval filtering — the agent could specify section: "90.325" as a filter when it knows the statute it needs.

Tradeoffs:

• ✅ Enables schema-level filtering by statute number — the agent can guarantee retrieval of a specific section when it knows which one it needs
• ✅ Section metadata (title, number, jurisdiction) becomes a first-class retrieval signal, not just buried text
• ✅ Better foundation for future multi-jurisdiction support (see Multi-bucket / multi-jurisdiction datastore support across GCS ingestion pipeline #353)
• ⚠️ Requires datastore schema changes and a more complex ingestion pipeline
• ⚠️ Vertex AI Search structured-data ingestion has different configuration than the current unstructured ingestion — significant rework
• ⚠️ The agent would need to be updated to pass schema filters in some cases

Option C: Redundant targeted retrieval (retrieval-time mitigation, no corpus change)

Keep the corpus as-is but add a second retrieval pass for high-risk statutes. When the agent's query matches a known pattern (DV scenario → try ORS 90.325; abandoned-property scenario → try ORS 90.425), issue an additional targeted query using the exact statute citation as the query string.

How: Add a post-processing step in langchain_tools.py that detects trigger patterns in the original query and issues supplemental targeted queries with statute-specific strings. Alternatively, add new specialized tools that always query with statute-specific strings for known problem areas.

Tradeoffs:

• ✅ No corpus changes, no datastore rebuild — lowest implementation risk
• ✅ Can be shipped incrementally alongside the existing STOPGAP safety net
• ✅ Allows the STOPGAPs to be removed once the targeted retrieval is validated
• ⚠️ Doesn't fix the underlying chunking problem — other statutes with the same issue will surface the same way in future evaluations
• ⚠️ Requires maintaining a list of "known-problematic" statutes and their trigger patterns — a different kind of maintenance burden
• ⚠️ Adds latency (additional retrieval calls) and complexity to the tool layer

---

Recommendation

Option A is the most direct fix with the best cost/benefit ratio for the near term. It addresses the root cause for all ORS sections (not just the two with STOPGAPs), fits within the existing pipeline, and removes the need for STOPGAP maintenance. The cross-section context loss is a real tradeoff but can be measured empirically — the eval suite can detect regressions on cross-section questions.

Option B becomes worth the investment when multi-jurisdiction support (#353) or schema-level filtering becomes a priority. Option C is a reasonable stepping stone if A or B requires a long runway.

---

Acceptance criteria

• ORS 90.325(3)(b) verbatim text retrieved on ≥ 3/4 of real eval queries at production params
• ORS 90.425(6)(b) verbatim text retrieved on ≥ 3/4 of real eval queries at production params
• Both STOPGAP blocks removed from system_prompt.md
• S1 (abandoned property) eval mean holds at ≥ 0.90 after STOPGAP removal
• S3 (DV victim damage) eval mean holds at ≥ 0.95 after STOPGAP removal
• No regressions on other scenarios (S0, S2, S4–S6) vs. the pre-change baseline

Related

• Revision-controlled corpus-cfg.TOML for RAG deployment + chunking/granularity flags #354 Revision-controlled corpus-cfg.TOML for RAG deployment + chunking/granularity flags
• Tracking GCP RAG Data Stores #276 Tracking GCP RAG Data Stores
• Multi-bucket / multi-jurisdiction datastore support across GCS ingestion pipeline #353 Multi-bucket / multi-jurisdiction datastore support