Skip to content

Commit f2770ce

Browse files
lalitdotdevlalitdotdev
committed
feat: add project roadmap and development phase documentation to todos.md
1 parent 82d45a6 commit f2770ce

1 file changed

Lines changed: 149 additions & 0 deletions

File tree

‎todos.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Wayfarer — TODO
2+
3+
Improvement roadmap for the multi-agent pipeline (`main.py`) and supporting modules.
4+
Do not touch `app.py` UI/CSS while working through these — backend only.
5+
6+
Work top to bottom. Each phase should leave the pipeline runnable end-to-end before moving to the next.
7+
8+
---
9+
10+
## Phase 1 — Fix silent failure in `planner_agent`
11+
12+
- [ ] Replace raw `json.loads` + try/except-default-to-"unknown" with Pydantic schema validation for planner output
13+
- [ ] On parse/validation failure, retry the LLM call once with an explicit "return valid JSON matching this schema" correction prompt
14+
- [ ] If retry also fails, surface a clear error to the user instead of silently proceeding with `"unknown"` fields
15+
- [ ] Confirm `flight_agent`/`hotel_agent` no longer silently skip search due to swallowed parse errors
16+
17+
## Phase 2 — Add clarification interrupt
18+
19+
- [ ] Use LangGraph interrupt / human-in-the-loop pattern (Postgres checkpointing already supports this)
20+
- [ ] If `planner_agent` can't extract destination or dates, interrupt the graph and prompt the user for the missing field
21+
- [ ] Resume cleanly from checkpoint once user responds
22+
- [ ] Remove the old silent default-to-"unknown" fallback path once this is in place
23+
24+
## Phase 3 — RAG layer: Destination Guide Agent
25+
26+
- [ ] Set up Pinecone index for destination guide content (visa requirements, safety info, local tips)
27+
- [ ] Seed starter dataset for 10-15 popular destinations, chunked appropriately
28+
- [ ] Build new agent node that retrieves via hybrid search (metadata filter by destination + vector similarity)
29+
- [ ] Run this agent in parallel with `flight_agent`/`hotel_agent`
30+
- [ ] Feed retrieved guide content into `itinerary_agent`; itinerary should cite it in the final output
31+
32+
## Phase 4 — Structured `hotel_tool.py`
33+
34+
- [ ] Build `hotel_tool.py` mirroring `flight_tool.py`'s structured output pattern
35+
- [ ] Return structured hotel objects: name, price, rating, location, booking link
36+
- [ ] Replace raw Tavily search text currently passed into the itinerary prompt
37+
38+
## Phase 5 — Budget-aware planning
39+
40+
- [ ] Add `budget` field to `TravelState`
41+
- [ ] Parse budget out of queries in `planner_agent` (e.g. "under ₹2 lakhs")
42+
- [ ] Have `flight_agent`/`hotel_agent` filter/rank results against budget
43+
- [ ] Have `itinerary_agent` explicitly reference budget tradeoffs in final output
44+
45+
## Phase 6 — Observability
46+
47+
- [ ] Extend existing `llm_calls` counter into per-node tracing: latency + estimated token cost per agent
48+
- [ ] Log traces to Postgres
49+
- [ ] Surface a summary (total cost, total time, calls per agent) in the final output returned to the frontend
50+
51+
## Phase 7 — Eval harness
52+
53+
- [ ] Create `tests/eval_harness.py`
54+
- [ ] 20-25 golden test queries covering:
55+
- [ ] Router intent classification accuracy
56+
- [ ] Planner field-extraction accuracy (correct IATA codes/dates)
57+
- [ ] Itinerary faithfulness (LLM-as-judge check against retrieved flight/hotel/guide data)
58+
- [ ] Output pass/fail report with per-category scores, runnable via a single command
59+
60+
---
61+
62+
## Nice-to-have (after Phases 1-7)
63+
64+
- [ ] Multi-city trip support (currently single dep/arr IATA pair only)
65+
- [ ] Router/supervisor override — "just flights, skip hotels" conditional routing
66+
- [ ] Booking/save/compare flow for multiple itinerary variants
67+
68+
---
69+
70+
## Priority if time-constrained
71+
72+
If applying to jobs soon, do **Phase 1, 2, and 6 first** — smallest changes, highest "found and fixed a real reliability bug" interview value. Phase 3 (RAG) is the biggest lift but closes the specific gap of Wayfarer having no retrieval component.
73+
74+
# BREAKDOWN OF PHASES
75+
76+
`````
77+
I'm working on Wayfarer, a multi-agent travel planning system at
78+
github.com/lalitdotdev/wayfarer-multiagent. It uses LangGraph with
79+
Postgres checkpointing, orchestrating: Router → Planner → (Flight ∥ Hotel,
80+
parallel) → Itinerary → Final, built on Groq/LLaMA 3.3 70B, Tavily Search,
81+
and AviationStack. Frontend is a custom-styled Streamlit app (app.py) —
82+
do not touch app.py's UI/CSS, only backend logic in main.py and new files.
83+
84+
Work through these phases in order. After each phase, run the existing
85+
tests, confirm the pipeline still runs end-to-end on a sample query, and
86+
summarize what changed before moving to the next phase.
87+
88+
PHASE 1 — Fix the silent failure in planner_agent
89+
Currently, if the LLM's JSON response fails to parse in planner_agent,
90+
fields silently default to "unknown," and flight_agent/hotel_agent then
91+
skip search entirely with no user-visible error. Fix this:
92+
- Replace raw json.loads + try/except-default with Pydantic schema
93+
validation for the planner's structured output.
94+
- On parse/validation failure, retry the LLM call once with an explicit
95+
"return valid JSON matching this schema" correction prompt before
96+
falling back.
97+
- If it still fails after retry, surface a clear error to the user
98+
instead of silently proceeding with "unknown" fields.
99+
100+
PHASE 2 — Add a clarification interrupt
101+
The graph already has Postgres checkpointing, so use LangGraph's
102+
interrupt/human-in-the-loop pattern: if planner_agent can't extract a
103+
required field (destination, dates), interrupt the graph and ask the
104+
user for the missing info instead of defaulting to "unknown" and
105+
silently skipping downstream agents. Resume cleanly from the checkpoint
106+
once the user responds.
107+
108+
PHASE 3 — Add a RAG layer: Destination Guide Agent
109+
Add a new agent node that retrieves from a Pinecone vector index of
110+
destination guide content (visa requirements, safety info, local tips —
111+
you can seed this with a small starter dataset for 10-15 popular
112+
destinations, chunked appropriately). This agent runs in parallel with
113+
flight_agent/hotel_agent and its output feeds into itinerary_agent,
114+
which should cite the retrieved guide content in the final itinerary.
115+
Use hybrid search (metadata filter by destination + vector similarity)
116+
rather than naive top-k.
117+
118+
PHASE 4 — Structured hotel_tool.py
119+
Currently hotel data flows into the itinerary prompt as raw Tavily
120+
search text while flight data goes through flight_tool.py with
121+
structured output. Build hotel_tool.py mirroring flight_tool.py's
122+
pattern — structured hotel objects (name, price, rating, location,
123+
booking link) instead of raw search text.
124+
125+
PHASE 5 — Budget field
126+
Add a budget field to TravelState. Parse it out in planner_agent from
127+
queries like "under ₹2 lakhs." Have flight_agent and hotel_agent filter/
128+
rank results against it, and have itinerary_agent explicitly reference
129+
budget tradeoffs in the final output.
130+
131+
PHASE 6 — Observability
132+
Extend the existing llm_calls counter into real per-node tracing:
133+
latency and estimated token cost per agent node, logged to Postgres.
134+
Surface a summary (total cost, total time, calls per agent) in the
135+
final output returned to the frontend.
136+
137+
PHASE 7 — Eval harness
138+
Create a standalone eval script (tests/eval_harness.py) with 20-25
139+
golden test queries covering: router intent classification accuracy,
140+
planner field-extraction accuracy (correct IATA codes/dates), and
141+
itinerary faithfulness (does the final itinerary actually reflect the
142+
retrieved flight/hotel/guide data, checked via a separate LLM-as-judge
143+
call). Output a simple pass/fail report with per-category scores,
144+
runnable via a single command.
145+
146+
For each phase, tell me which files you're changing before you change
147+
them. Keep phases 1-2 minimal and surgical — don't refactor working code
148+
beyond what's needed to fix the described issue.````
149+
`````

0 commit comments

Comments
 (0)