AlphaSignal

AlphaSignal is a student project built to explore how a production style AI system can be designed around financial information retrieval.

The core idea is simple: ingest company news and financial snapshot data, index it for hybrid retrieval, and let a user query it through a Telegram bot that uses GenAI and agentic workflows to return grounded answers.

This project was also an attempt to move beyond notebook style AI projects and build something closer to an operable system with:

• Dockerized services
• scheduled pipelines
• persistent storage
• Hybrid search (options for simple vector search and simple BM25 search also) (used RRF, will include reranking in future iteration)
• Redis caching (semantic caching on query)
• RAG
• LangGraph based agent workflows
• tracing and debugging with LangSmith (was very useful)

It is not presented as a finished production product. It is a serious learning project focused on system design, ML infrastructure, GenAI integration, and debugging real data pipelines.

Problem Statement

Company research is fragmented across market data platforms, news sources, and dashboards. Even when the data exists, it is not easy to ask a natural language question and get a concise, grounded answer with sources.

This project tries to reduce that friction by building a system that:

• fetches financial and news data for tracked companies
• stores and indexes that data
• supports hybrid semantic retrieval
• uses RAG to generate grounded answers
• pushes scheduled updates for the watchlist instead of waiting only for user queries (just slightly after the market starts for fresh news and financial analysis daily)
• exposes the whole flow through Telegram

What It Solves

The system is designed for a simple but practical use case:

• receive timely updates on a watchlist
• ask for a ticker on demand
• fetch fresh data if the ticker is missing
• get a summary based on retrieved company data instead of an ungrounded LLM response

In short, it is a watchlist intelligence assistant with an AI retrieval and summarization layer.

One part that I think is important is that the system is not only reactive. Getting an answer to a question is relatively straightforward in many RAG systems. In this project, I also wanted the system to behave proactively by sending daily updates for all watchlist stocks. That is why the Airflow layer is important: scheduled DAGs keep the data fresh, and the Telegram bot can push a morning pre-session brief without waiting for the user to ask first.

Why I Built It This Way

I wanted this project to teach me more than just prompt engineering.

I chose Dockerization because I wanted:

• an easier path toward deployment later
• a better understanding of service boundaries
• hands on experience with a microservices style architecture
• reproducible local setup across backend, database, cache, scheduler, and search engine

I chose Airflow, OpenSearch, Redis, LangGraph, and LangSmith because I wanted to understand how modern AI systems are actually glued together in practice, not only how to call an LLM API.

Tech Stack

• FastAPI for the backend service
• PostgreSQL for durable structured storage
• OpenSearch for keyword + vector retrieval
• Redis for semantic RAG caching
• Apache Airflow for scheduled and on demand workflows
• FastEmbed for dense embeddings (downloading sentence transformer was heavy across so many containers)
• Groq + LangChain for LLM based answer generation
• LangGraph for the Telegram workflow agent
• LangSmith for tracing and debugging the GenAI pipeline
• Docker Compose for multi service local orchestration

Architecture

Main Services

• backend: owns search, hybrid retrieval, caching, and RAG
• postgres: stores tickers, documents, metadata, and watchlist state
• opensearch: stores indexed documents and dense vectors
• redis: stores semantic cache entries for repeated RAG queries
• airflow-scheduler and airflow-webserver: run and manage DAGs
• telegram-poller: receives Telegram messages and routes them through a LangGraph agent

Data Flow

1. Airflow ingests news from Alpha Vantage and financial snapshots from Yahoo Finance
2. Raw documents are stored in PostgreSQL
3. New documents are embedded and indexed into OpenSearch
4. The backend performs hybrid retrieval over OpenSearch
5. Retrieved documents are passed into an LLM through a RAG prompt
6. Redis caches semantically similar RAG answers
7. The Telegram bot uses a LangGraph agent to route requests and trigger refresh workflows when needed

Architecture Diagram Placeholder

Telegram Screenshots Placeholder

Airflow Dashboard Screenshot Placeholder

GenAI in This Project

GenAI is not just used as a final text generator. It is part of the system design in multiple places.

1. Embeddings

The project uses sentence-transformers/all-MiniLM-L6-v2 through fastembed to create dense vectors.

These embeddings are used for:

• vector retrieval in OpenSearch
• semantic matching in Redis cache
• improving search beyond exact keyword overlap

2. Retrieval-Augmented Generation

The main /search/rag route performs:

• hybrid retrieval
• context building
• prompt construction
• grounded LLM answering

The LLM is instructed to answer only from retrieved context and cite sources. This reduces hallucination risk and makes the answer more useful for company updates.

3. Semantic Caching

Redis stores previous RAG answers along with the embedded query. If a later question is semantically similar (kept the threshold quite high (.92) to avoid too many false positives) and matches the same ticker, the system can reuse the earlier answer instead of repeating the full LLM call.

This reduces:

• latency
• repeated model usage
• unnecessary cost

Agentic AI in This Project

The Telegram workflow is built as a small LangGraph agent system.

The point was not to make a fully autonomous agent. The point was to use an agent framework where it made sense: workflow routing and system interaction.

What the agent does

The LangGraph layer:

• interprets Telegram messages
• distinguishes watchlist commands from ticker queries
• routes /add, /remove, /watchlist, and /daily
• triggers Airflow refresh DAGs when a ticker has no indexed data
• rechecks for results and supports delayed follow-up messages

Why LangGraph was useful

Without LangGraph, the Telegram layer would become a growing set of conditionals mixed with infrastructure calls.

LangGraph helped model the interaction as a small workflow graph while still keeping the actual search and RAG logic inside the backend, where it belongs.

That was an important design decision:

• backend owns retrieval and RAG
• Airflow owns ingestion and indexing
• LangGraph owns intent routing and workflow transitions

Airflow Workflows

news_ingest

• fetches company news and sentiment from Alpha Vantage
• supports both scheduled watchlist runs and ticker specific manual runs through dag_run.conf
• stores ticker-specific sentiment and relevance in PostgreSQL

yfinance_ingest

• fetches compact financial snapshots from Yahoo Finance
• stores valuation and market metrics as documents
• also supports scheduled watchlist runs and ticker specific refreshes

index_documents

• reads unindexed documents from PostgreSQL
• generates embeddings
• pushes documents and vectors into OpenSearch
• marks documents as indexed after success

telegram_daily_update

• sends a daily watchlist summary to Telegram
• runs separately from the query flow

This was an important part of the project design. The system is not only a search interface over indexed documents. It also behaves like a scheduled monitoring system for the watchlist. The idea is that the user receives a concise analysis update before starting the trading day, so the system can act as a pre-session briefing tool rather than only a chatbot.

Search and Retrieval Design

The retrieval layer uses hybrid search:

• BM25 for lexical relevance
• vector search for semantic relevance
• reciprocal rank fusion to combine the two

I kept one important heuristic in the retrieval layer: when possible, include at least one recent yfinance financial snapshot together with the news results. This helps the final answer include both numerical context and recent events. Without this, sometimes all the top k docs fetched were from alpha vantage, completely missing the financial analysis part.

LangSmith for Debugging and Monitoring (was very helpful)

LangSmith was used as the tracing layer for the GenAI workflow.

This was useful because many failures did not come from the model alone. They came from interactions between:

• retrieval
• prompt construction
• stale cache entries
• empty context
• differences between backend behavior and Telegram behavior

LangSmith helped inspect:

• the exact prompt sent to the model
• whether the LLM received useful context
• whether a failure was in retrieval or generation
• whether bot behavior matched direct backend behavior

This was especially helpful during debugging when OpenSearch already had documents but the Telegram flow was still reaching the model with bad or empty context.

The project enables LangSmith through standard LangChain tracing environment variables such as:

• LANGCHAIN_API_KEY
• LANGCHAIN_TRACING_V2=true
• LANGCHAIN_PROJECT=alphasignal

LangSmith Dashboard Screenshot Placeholder

Repository Structure

.
├── airflow/
│   ├── Dockerfile
│   └── dags/
│       ├── news_dag.py
│       ├── yfinance_dag.py
│       ├── index_documents_dag.py
│       ├── telegram_agent.py
│       └── langgraph_agents.py
├── backend/
│   ├── Dockerfile
│   ├── main.py
│   ├── pyproject.toml
│   └── routers/
│       └── search.py
├── db/
│   ├── init.sql
│   └── migrate_existing.sql
└── docker-compose.yml

Running the Project

Environment variables

Create a root .env file with values such as:

GROQ_API_KEY=...
ALPHAVANTAGE_API_KEY=...
AIRFLOW__CORE__FERNET_KEY=...
TELEGRAM_BOT_TOKEN=...
TELEGRAM_CHAT_ID=...
LANGCHAIN_API_KEY=

Start the stack

docker compose up -d --build

Apply live schema migration if needed

docker compose exec -T postgres psql -U admin -d alpha < db/migrate_existing.sql

Useful endpoints

• FastAPI backend: http://localhost:8000
• Airflow UI: http://localhost:8080
• OpenSearch: http://localhost:9200

Example bot commands

/add AAPL
/remove AAPL
/watchlist
/daily
MSFT

Technical Choices

Why Docker Compose

I wanted one reproducible environment that could run the full stack locally and also serve as a base for future deployment work. It also helped me think about the project as a microservices system rather than one Python process doing everything.

Why OpenSearch

Because the project needed both:

• lexical retrieval
• semantic vector retrieval

and I wanted those capabilities in one search layer.

Why Redis

Because RAG answers are expensive and often repetitive. Semantic caching is a practical system optimization, not just an extra feature.

Why Airflow

Because this project has real pipeline steps with timing and dependency concerns. I wanted a scheduler and orchestration layer that made those flows explicit.

Why LangGraph

Because the Telegram interaction is naturally a workflow problem. I wanted to use an agent framework in a bounded way rather than writing a growing command router by hand.

Key Learnings

• Building AI systems is often more about data pipelines and debugging than model calls
• Retrieval quality matters more than prompt cleverness for grounded company updates
• Empty or stale cache entries can silently damage RAG quality
• Agent layers should orchestrate workflows, not duplicate backend logic
• Docker build speed depends heavily on dependency-layer design
• Observability tools like LangSmith become much more useful once multiple AI and infrastructure components interact

Difficult Parts

• schema drift between code and database
• empty context reaching the LLM
• cache poisoning from poor RAG responses
• Airflow refreshes initially running for all tickers instead of one
• bot behavior diverging from direct backend behavior
• separating watchlist membership from general ticker existence

These were useful problems because they exposed the difference between "the code runs" and "the system behaves correctly end to end."

Limitations

• This is not a trading system
• It is not a backtesting or portfolio-evaluation platform
• It depends on external APIs and free-tier constraints
• Good free APIs for Indian stocks were hard to find with generous limits, so the current pipeline is designed primarily around US stocks
• The daily summary is intentionally simple
• The LangGraph layer is deliberately lightweight, not a full autonomous analyst system

Final Note

This project is best understood as a serious engineering and AI systems exercise.

It was my attempt to build something that combines:

• data engineering
• search infrastructure
• vector retrieval
• GenAI
• agent workflows
• containerized deployment thinking

in one coherent project.

A large part of the learning came from debugging failure modes across the full system, not just from getting a model to produce text.