Roadmap to v1.0.0

[TonsOfFun]

Roadmap to v1.0.0

🚀 Core Providers

Feature	Description	Status	Notes / Dependencies
TransformerProvider	Support for transformer-based models (e.g. HuggingFace, OpenAI, etc.)	Pending / In Progress	API interface, authentication, rate-limiting, cost management
ONNXProvider	Run ONNX models locally / in production	Planned	Model conversion, resource usage, deployment concerns
RubyLLMProvider	Native Ruby-based LLMs or bindings (if available)	Planned	Evaluate existing gems; performance benchmarks

---

⚙ Prompt Management & Monitoring

Feature	Description	Status	Notes
Prompt Management	Versioned prompts, templating, prompt reuse, auditing	In Planning	Integration with UI / Rails admin, prompt history
Agent Monitoring	Metrics & logging for agent runs, failures, usage	In Planning	What metrics — latency, throughput, error types, cost?

---

🔍 Evaluation & Judging

Feature	Description	Status	Notes
Evaluations	Facility to run tests / benchmarks on agent output	Planned	Might need test harness, golden datasets
Judging	Automated or semi-automated evaluation of agent responses	Planned	Define judging criteria; human vs automated

---

🧠 Agent Functionality & Context Management

Feature	Description	Status	Notes
Context	Long-term memory, prior conversation, retrieval-augmented context	Under Design	Vector embeddings; storage; similarity search
Message	Unified message models for communications between user/agent/tool	In Progress	Schema for message types; consistency
Actions	Built-in tools for agents to act (APIs, database, external systems)	Ongoing	Secure execution; sandboxing; extensibility

---

🧱 “Solid Agent” Framework

Feature	Description	Status	Notes
Solid Agent	A stable, opinionated agent model / pattern that handles common cases well	Conceptual	Provide defaults; built-in best practices; configuration over code

---

📅 Suggested Timeline

Phase	Approximate Timeframe	Focus Areas
Phase 1	Week 1-2	TransformerProvider + Prompt Management, basic Context and Message schema, first Monitoring metrics
Phase 2	Week 3-4	ONNXProvider, RubyLLMProvider, Actions & Evaluations, extending Monitoring
Phase 3	Week 5-6	Judging, Solid Agent scaffold, polishing stability, documentation, UX/admin for prompt/versioned prompts

---

✅ Other Considerations

• Security & Privacy: Ensuring data handling, prompt outputs, and user info are secure.
• Cost & Infrastructure: Model hosting (especially transformer / ONNX), compute cost, scaling.
• Extensibility / Plugin System: Allow 3rd-party tools, providers, and prompts to integrate smoothly.
• Documentation, Examples & Community Feedback: Validate that features solve real user needs.

[sirwolfgang]

For Metrics/Monitoring, I haven't set it up, but NR is now offering this if you want to take a look at what they do: https://docs.newrelic.com/docs/ai-monitoring/intro-to-ai-monitoring/

[richardchen874-sys]

The provider roadmap looks important for a production agent framework. Once agents have memory, tools, retries, and external actions, model selection becomes part of the runtime policy, not just a config option.

I’d probably make these controls visible early:

• per-agent token and cost tracking
• provider capability profiles
• cheap model for routine memory/tool steps
• stronger model for planning or final decisions
• fallback / downgrade rules when a provider fails or gets expensive

I'm building a private-beta OpenAI-compatible API aggregation layer focused on Chinese models like DeepSeek / Qwen / GLM-style providers, using official-provider keys. If you want to benchmark lower-cost providers for ActiveAgent runs, I’d be happy to share some test credits.

Curious: do you expect cost management to live in the provider layer, prompt layer, or agent runtime layer?