diff --git a/README.md b/README.md index 2142a8b..8c0e539 100644 --- a/README.md +++ b/README.md @@ -153,14 +153,14 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR *Rotating pinned items: the most-discussed agentic & AI-engineering resources of the current cycle. Refreshed regularly โ€” see [CONTRIBUTING.md](./CONTRIBUTING.md) for nomination criteria.* -- โญ ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic (2024). The most-cited reference for agent design patterns (augmented LLM, prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser, autonomous agents). Start here before any other agent reading. -- โญ ๐Ÿ†“ [**How we built our multi-agent research system**](https://www.anthropic.com/engineering/built-multi-agent-research-system) โ€” Anthropic (2025). Production retrospective on Claude's multi-agent research mode: orchestrator/subagent split, prompt engineering for agents, evaluation and failure modes. - โญ ๐Ÿ†“ [**A practical guide to building agents**](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf) โ€” OpenAI (2025). 30-page PDF covering when (and when not) to build agents, tool design, guardrails, and human-in-the-loop patterns. -- โญ ๐Ÿ†“ [**The bitter lesson of AI agents**](https://lucumr.pocoo.org/2025/6/12/agentic-coding/) / [**Agentic Coding: The Future of Software Development with Agents**](https://lucumr.pocoo.org/) โ€” Armin Ronacher (2025). Widely-shared essays on what it actually feels like to ship with agentic coding tools day-to-day. +- ๐Ÿ†“ [**AGENTS.md**](https://agents.md/) โ€” Community standard (2025) for per-repo agent instructions, now read by Claude Code, Codex, Aider, Cursor, Cline, Windsurf and others. +- โญ ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic (2024). The most-cited reference for agent design patterns (augmented LLM, prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser, autonomous agents). Start here before any other agent reading. - ๐Ÿ†“ [**Claude Code: Best practices for agentic coding**](https://www.anthropic.com/engineering/claude-code-best-practices) โ€” Anthropic (2025). CLAUDE.md, slash-commands, headless mode, custom permissions โ€” the canonical how-to-use-Claude-Code reference. - ๐Ÿ†“ [**How to build an agent**](https://ampcode.com/how-to-build-an-agent) โ€” Thorsten Ball / Amp (2025). Viral step-by-step implementation of a tool-using coding agent in ~400 lines of Go, demystifying "what is an agent" in code. +- โญ ๐Ÿ†“ [**How we built our multi-agent research system**](https://www.anthropic.com/engineering/built-multi-agent-research-system) โ€” Anthropic (2025). Production retrospective on Claude's multi-agent research mode: orchestrator/subagent split, prompt engineering for agents, evaluation and failure modes. +- โญ ๐Ÿ†“ [**The bitter lesson of AI agents**](https://lucumr.pocoo.org/2025/6/12/agentic-coding/) / [**Agentic Coding: The Future of Software Development with Agents**](https://lucumr.pocoo.org/) โ€” Armin Ronacher (2025). Widely-shared essays on what it actually feels like to ship with agentic coding tools day-to-day. - ๐Ÿ†“ [**The new code**](https://www.latent.space/p/new-code) โ€” Sean Grove / OpenAI on Latent Space (2025). Specs-as-code: the spec is the new artefact, models are the compiler. Heavily cited in the AGENTS.md / spec-kit discussion. -- ๐Ÿ†“ [**AGENTS.md**](https://agents.md/) โ€” Community standard (2025) for per-repo agent instructions, now read by Claude Code, Codex, Aider, Cursor, Cline, Windsurf and others. @@ -172,43 +172,43 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR - โญ ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic. The reference taxonomy of agent design patterns (workflows vs. agents). - โญ ๐Ÿ†“ [**Claude Code: Best practices for agentic coding**](https://www.anthropic.com/engineering/claude-code-best-practices) โ€” Anthropic. CLAUDE.md, tools, slash-commands, headless mode. -- ๐Ÿ†“ [**How to build an agent**](https://ampcode.com/how-to-build-an-agent) โ€” Thorsten Ball. A working coding agent in ~400 lines; the clearest "agents are not magic" walkthrough. - ๐Ÿ†“ [**Here's how I use LLMs to help me write code**](https://simonwillison.net/2025/Mar/11/using-llms-for-code/) โ€” Simon Willison. Grounded, practice-first account of daily LLM-assisted development. +- ๐Ÿ†“ [**How to build an agent**](https://ampcode.com/how-to-build-an-agent) โ€” Thorsten Ball. A working coding agent in ~400 lines; the clearest "agents are not magic" walkthrough. ### T2 ยท Spec-Driven Development & Context Engineering -- โญ ๐Ÿ†“ [**The new code**](https://www.latent.space/p/new-code) โ€” Sean Grove (OpenAI) on Latent Space. The canonical "specs are the new code" essay. - ๐Ÿ†“ [**AGENTS.md**](https://agents.md/) โ€” Community standard for per-repo agent instructions. - ๐Ÿ†“ [**spec-kit**](https://github.com/github/spec-kit) โ€” GitHub's toolkit and essay set on spec-driven development with coding agents. +- โญ ๐Ÿ†“ [**The new code**](https://www.latent.space/p/new-code) โ€” Sean Grove (OpenAI) on Latent Space. The canonical "specs are the new code" essay. - ๐Ÿ†“ [**The rise of "context engineering"**](https://blog.langchain.com/the-rise-of-context-engineering/) โ€” LangChain. Why prompt engineering became context engineering. ### T6 ยท LLM Application Architecture & System Design - โญ ๐Ÿ“˜ ๐Ÿ’ฐ [**AI Engineering**](https://www.oreilly.com/library/view/ai-engineering/9781098166298/) โ€” Chip Huyen (O'Reilly, 2025). The textbook for building LLM applications end-to-end. -- โญ ๐Ÿ†“ [**Patterns for Building LLM-based Systems & Products**](https://eugeneyan.com/writing/llm-patterns/) โ€” Eugene Yan. Evaluation, RAG, fine-tuning, caching, guardrails, defensive UX, collecting feedback โ€” the reference pattern catalogue. - ๐Ÿ†“ [**Emerging Architectures for LLM Applications**](https://a16z.com/emerging-architectures-for-llm-applications/) โ€” a16z. The widely-shared reference diagram for the LLM app stack. +- โญ ๐Ÿ†“ [**Patterns for Building LLM-based Systems & Products**](https://eugeneyan.com/writing/llm-patterns/) โ€” Eugene Yan. Evaluation, RAG, fine-tuning, caching, guardrails, defensive UX, collecting feedback โ€” the reference pattern catalogue. - ๐Ÿ†“ [**What We Learned from a Year of Building with LLMs**](https://applied-llms.org/) โ€” Yan, Bensal, Bhawal, Husain, Shankar (2024). Tactical, operational, and strategic lessons distilled from shipping. ### T7 ยท Prompt Engineering +- ๐Ÿ†“ [**Anthropic: Prompt engineering overview**](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) โ€” Anthropic's practical guide for Claude. +- ๐Ÿ†“ [**OpenAI: Prompt engineering**](https://platform.openai.com/docs/guides/prompt-engineering) โ€” OpenAI official guide. - โญ ๐Ÿ†“ [**Prompt Engineering**](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/) โ€” Lilian Weng (OpenAI). The systematic taxonomy. - ๐Ÿ†“ [**Prompt Engineering Guide**](https://www.promptingguide.ai/) โ€” DAIR.AI. Continuously updated, with per-technique deep-dives. -- ๐Ÿ†“ [**OpenAI: Prompt engineering**](https://platform.openai.com/docs/guides/prompt-engineering) โ€” OpenAI official guide. -- ๐Ÿ†“ [**Anthropic: Prompt engineering overview**](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) โ€” Anthropic's practical guide for Claude. ### T8 ยท Retrieval-Augmented Generation (RAG) -- โญ ๐Ÿ“„ ๐Ÿ†“ [**Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**](https://arxiv.org/abs/2005.11401) โ€” Lewis et al. (2020). The original RAG paper. - โญ ๐Ÿ†“ [**Advanced RAG Techniques**](https://www.pinecone.io/learn/advanced-rag-techniques/) / [**Pinecone Learn**](https://www.pinecone.io/learn/) โ€” Pinecone. The hub for RAG primers and patterns. -- ๐Ÿ†“ [**Retrieval-Augmented Generation for LLMs: A Survey**](https://arxiv.org/abs/2312.10997) โ€” Gao et al. (2023). The reference survey. - ๐Ÿ†“ [**RAG is more than just embedding search**](https://jxnl.co/writing/2024/06/11/rag-is-more-than-just-embedding-search/) โ€” Jason Liu. Systems-view RAG: query understanding, tool routing, evaluation. +- โญ ๐Ÿ“„ ๐Ÿ†“ [**Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks**](https://arxiv.org/abs/2005.11401) โ€” Lewis et al. (2020). The original RAG paper. +- ๐Ÿ†“ [**Retrieval-Augmented Generation for LLMs: A Survey**](https://arxiv.org/abs/2312.10997) โ€” Gao et al. (2023). The reference survey. ### T10 ยท Tool Use, Function Calling & MCP +- ๐Ÿ†“ [**Function calling guide**](https://platform.openai.com/docs/guides/function-calling) โ€” OpenAI. The canonical reference for structured tool calls. - โญ ๐Ÿ†“ [**Introducing the Model Context Protocol**](https://www.anthropic.com/news/model-context-protocol) โ€” Anthropic (2024). The canonical introduction to MCP. - โญ ๐Ÿ†“ [**Model Context Protocol โ€” Specification**](https://modelcontextprotocol.io/) โ€” Open protocol docs and SDKs. - ๐Ÿ“„ ๐Ÿ†“ [**Toolformer: Language Models Can Teach Themselves to Use Tools**](https://arxiv.org/abs/2302.04761) โ€” Schick et al. (2023). The foundational tool-use paper. -- ๐Ÿ†“ [**Function calling guide**](https://platform.openai.com/docs/guides/function-calling) โ€” OpenAI. The canonical reference for structured tool calls. ### T11 ยท Orchestration, Planning & Design Patterns @@ -219,10 +219,10 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T13 ยท Evaluation & Testing -- โญ ๐Ÿ†“ [**Your AI Product Needs Evals**](https://hamel.dev/blog/posts/evals/) โ€” Hamel Husain. The most-cited essay on why and how to build evals for LLM products. -- ๐Ÿ†“ [**Task-Specific LLM Evals that Do & Don't Work**](https://eugeneyan.com/writing/evals/) โ€” Eugene Yan. A pragmatic survey of eval techniques per task type. - ๐Ÿ“„ ๐Ÿ†“ [**Judging LLM-as-a-Judge**](https://arxiv.org/abs/2306.05685) โ€” Zheng et al. (2023). The foundational LLM-as-judge paper (MT-Bench, Chatbot Arena). +- ๐Ÿ†“ [**Task-Specific LLM Evals that Do & Don't Work**](https://eugeneyan.com/writing/evals/) โ€” Eugene Yan. A pragmatic survey of eval techniques per task type. - ๐Ÿ†“ [**Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences**](https://arxiv.org/abs/2404.12272) โ€” Shankar et al. (2024). How to make LLM-judges trustworthy. +- โญ ๐Ÿ†“ [**Your AI Product Needs Evals**](https://hamel.dev/blog/posts/evals/) โ€” Hamel Husain. The most-cited essay on why and how to build evals for LLM products. @@ -263,18 +263,18 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR *Discords, Slacks, forums, and meetups where practitioners gather.* -- ๐Ÿ†“ [**MLOps Community**](https://mlops.community/) โ€” Slack + podcast + meetups; the biggest practitioner community at the ops/engineering intersection. Active agent and LLM-ops channels. +- ๐Ÿ†“ [**AI Dev Board**](https://aidevboard.com/) โ€” Community-curated hub for AI engineering resources and discussions. +- ๐Ÿ†“ [**AI Engineer World's Fair / Latent Space Discord**](https://discord.gg/latent-space) โ€” Practitioner community anchoring the AI Engineer conference series. +- ๐Ÿ†“ [**Anthropic Discord**](https://www.anthropic.com/discord) โ€” Official Claude / Claude Code / MCP community. +- ๐Ÿ†“ [**Cursor Community Forum**](https://forum.cursor.com/) โ€” User-driven forum for Cursor rules, MCP, and workflows. +- ๐Ÿ†“ [**EleutherAI Discord**](https://discord.gg/zBGx3azzUn) โ€” Open research community; strong training/interpretability discussion. +- ๐Ÿ†“ [**Hacker News**](https://news.ycombinator.com/) โ€” Filter for "LLM", "agent", "Claude", "Cursor" โ€” where engineering-side essays trend. +- ๐Ÿ†“ [**Hugging Face Discord & Forums**](https://huggingface.co/join/discord) โ€” Transformers, TRL, PEFT, model-hub discussions. - ๐Ÿ†“ [**LangChain Discord**](https://discord.gg/langchain) โ€” Heavy day-to-day Q&A on agent orchestration, RAG, evaluation, MCP. - ๐Ÿ†“ [**LlamaIndex Discord**](https://discord.gg/dGcwcsnxhU) โ€” RAG-centric builder community with active reference-impl discussion. +- ๐Ÿ†“ [**MLOps Community**](https://mlops.community/) โ€” Slack + podcast + meetups; the biggest practitioner community at the ops/engineering intersection. Active agent and LLM-ops channels. - ๐Ÿ†“ [**r/LocalLLaMA**](https://www.reddit.com/r/LocalLLaMA/) โ€” The definitive open-weights / local-inference forum; fastest signal for new models, quantisation, and serving. - ๐Ÿ†“ [**r/MachineLearning**](https://www.reddit.com/r/MachineLearning/) โ€” Academic and practitioner mix; where new papers and threads get dissected. -- ๐Ÿ†“ [**Hacker News**](https://news.ycombinator.com/) โ€” Filter for "LLM", "agent", "Claude", "Cursor" โ€” where engineering-side essays trend. -- ๐Ÿ†“ [**EleutherAI Discord**](https://discord.gg/zBGx3azzUn) โ€” Open research community; strong training/interpretability discussion. -- ๐Ÿ†“ [**Hugging Face Discord & Forums**](https://huggingface.co/join/discord) โ€” Transformers, TRL, PEFT, model-hub discussions. -- ๐Ÿ†“ [**AI Engineer World's Fair / Latent Space Discord**](https://discord.gg/latent-space) โ€” Practitioner community anchoring the AI Engineer conference series. -- ๐Ÿ†“ [**AI Dev Board**](https://aidevboard.com/) โ€” Community-curated hub for AI engineering resources and discussions. -- ๐Ÿ†“ [**Cursor Community Forum**](https://forum.cursor.com/) โ€” User-driven forum for Cursor rules, MCP, and workflows. -- ๐Ÿ†“ [**Anthropic Discord**](https://www.anthropic.com/discord) โ€” Official Claude / Claude Code / MCP community. @@ -285,8 +285,8 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T1 ยท Coding Agents & AI-Assisted Development - โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**AI Python for Beginners**](https://www.deeplearning.ai/short-courses/ai-python-for-beginners/) โ€” DeepLearning.AI (Andrew Ng). Gateway to AI-assisted coding. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Pair Programming with a Large Language Model**](https://www.deeplearning.ai/short-courses/pair-programming-llm/) โ€” DeepLearning.AI + Google. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**GitHub Copilot Fundamentals**](https://learn.microsoft.com/en-us/training/paths/copilot/) โ€” Microsoft Learn. Official training path. +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Pair Programming with a Large Language Model**](https://www.deeplearning.ai/short-courses/pair-programming-llm/) โ€” DeepLearning.AI + Google. ### T4 ยท SWE Benchmarks & Coding Evaluation @@ -296,14 +296,14 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T6 ยท LLM Application Architecture & System Design -- โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**LLM Bootcamp**](https://fullstackdeeplearning.com/llm-bootcamp/) โ€” Full Stack Deep Learning. Free 2-day bootcamp on building LLM apps end-to-end. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Building Systems with the ChatGPT API**](https://www.deeplearning.ai/short-courses/building-systems-with-chatgpt/) โ€” DeepLearning.AI + OpenAI. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**CS25: Transformers United**](https://web.stanford.edu/class/cs25/) โ€” Stanford. Seminal deep-dive seminar series. +- โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**LLM Bootcamp**](https://fullstackdeeplearning.com/llm-bootcamp/) โ€” Full Stack Deep Learning. Free 2-day bootcamp on building LLM apps end-to-end. ### T7 ยท Prompt Engineering -- โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**ChatGPT Prompt Engineering for Developers**](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) โ€” Andrew Ng & Isa Fulford (OpenAI). - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Anthropic Prompt Engineering Interactive Tutorial**](https://github.com/anthropics/prompt-eng-interactive-tutorial) โ€” Anthropic. Hands-on, notebook-based. +- โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**ChatGPT Prompt Engineering for Developers**](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/) โ€” Andrew Ng & Isa Fulford (OpenAI). - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Prompt Engineering Guide (DAIR.AI)**](https://www.promptingguide.ai/) โ€” Self-paced, continuously updated. ### T8 ยท Retrieval-Augmented Generation (RAG) @@ -316,38 +316,38 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T10 ยท Tool Use, Function Calling & MCP - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Functions, Tools and Agents with LangChain**](https://www.deeplearning.ai/short-courses/functions-tools-agents-langchain/) โ€” DeepLearning.AI + LangChain. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**MCP: Build Rich-Context AI Apps with Anthropic**](https://www.deeplearning.ai/short-courses/mcp-build-rich-context-ai-apps-with-anthropic/) โ€” DeepLearning.AI + Anthropic. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Introduction to MCP**](https://modelcontextprotocol.io/quickstart) โ€” Anthropic official quickstart. +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**MCP: Build Rich-Context AI Apps with Anthropic**](https://www.deeplearning.ai/short-courses/mcp-build-rich-context-ai-apps-with-anthropic/) โ€” DeepLearning.AI + Anthropic. ### T11 ยท Orchestration, Planning & Design Patterns -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**AI Agents in LangGraph**](https://www.deeplearning.ai/short-courses/ai-agents-in-langgraph/) โ€” DeepLearning.AI + LangChain. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**AI Agentic Design Patterns with AutoGen**](https://www.deeplearning.ai/short-courses/ai-agentic-design-patterns-with-autogen/) โ€” DeepLearning.AI + Microsoft. +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**AI Agents in LangGraph**](https://www.deeplearning.ai/short-courses/ai-agents-in-langgraph/) โ€” DeepLearning.AI + LangChain. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Hugging Face Agents Course**](https://huggingface.co/learn/agents-course) โ€” Hugging Face. Free, certifying course on agent fundamentals. ### T12 ยท Multi-Agent Systems +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Building Agentic RAG with LlamaIndex**](https://www.deeplearning.ai/short-courses/building-agentic-rag-with-llamaindex/) โ€” DeepLearning.AI + LlamaIndex. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Multi AI Agent Systems with crewAI**](https://www.deeplearning.ai/short-courses/multi-ai-agent-systems-with-crewai/) โ€” DeepLearning.AI + crewAI. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Practical Multi AI Agents and Advanced Use Cases with crewAI**](https://www.deeplearning.ai/short-courses/practical-multi-ai-agents-and-advanced-use-cases-with-crewai/) โ€” DeepLearning.AI. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Building Agentic RAG with LlamaIndex**](https://www.deeplearning.ai/short-courses/building-agentic-rag-with-llamaindex/) โ€” DeepLearning.AI + LlamaIndex. ### T13 ยท Evaluation & Testing - โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ’ฐ [**AI Evals For Engineers & PMs**](https://maven.com/parlance-labs/evals) โ€” Hamel Husain & Shreya Shankar on Maven. The industry-standard evals cohort course. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Quality and Safety for LLM Applications**](https://www.deeplearning.ai/short-courses/quality-safety-llm-applications/) โ€” DeepLearning.AI + WhyLabs. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Automated Testing for LLMOps**](https://www.deeplearning.ai/short-courses/automated-testing-llmops/) โ€” DeepLearning.AI + CircleCI. +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Quality and Safety for LLM Applications**](https://www.deeplearning.ai/short-courses/quality-safety-llm-applications/) โ€” DeepLearning.AI + WhyLabs. ### T14 ยท Observability, Tracing & Debugging -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**LLMOps**](https://www.deeplearning.ai/short-courses/llmops/) โ€” DeepLearning.AI + Google Cloud. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Evaluating LLMs with Arize**](https://arize.com/llm-evaluation/) โ€” Arize course hub. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**LangSmith Academy**](https://academy.langchain.com/) โ€” LangChain. Free self-paced LangSmith courses covering tracing and evals. +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**LLMOps**](https://www.deeplearning.ai/short-courses/llmops/) โ€” DeepLearning.AI + Google Cloud. ### T15 ยท Guardrails & Security +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Prompt Injection Attacks (Learn Prompting)**](https://learnprompting.org/docs/prompt_hacking/injection) โ€” Learn Prompting. Open course covering injection/jailbreak taxonomies. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Red Teaming LLM Applications**](https://www.deeplearning.ai/short-courses/red-teaming-llm-applications/) โ€” DeepLearning.AI + Giskard. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Safe and Reliable AI via Guardrails**](https://www.deeplearning.ai/short-courses/safe-and-reliable-ai-via-guardrails/) โ€” DeepLearning.AI + Guardrails AI. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Prompt Injection Attacks (Learn Prompting)**](https://learnprompting.org/docs/prompt_hacking/injection) โ€” Learn Prompting. Open course covering injection/jailbreak taxonomies. ### T16 ยท Safety, Alignment & Responsible AI @@ -358,14 +358,14 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T17 ยท Fine-tuning, Post-training & RLHF - โญ ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Finetuning Large Language Models**](https://www.deeplearning.ai/short-courses/finetuning-large-language-models/) โ€” DeepLearning.AI + Lamini. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Reinforcement Learning from Human Feedback**](https://www.deeplearning.ai/short-courses/reinforcement-learning-from-human-feedback/) โ€” DeepLearning.AI + Google Cloud. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Hugging Face NLP Course (incl. RLHF chapter)**](https://huggingface.co/learn/nlp-course) โ€” Hugging Face. +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Reinforcement Learning from Human Feedback**](https://www.deeplearning.ai/short-courses/reinforcement-learning-from-human-feedback/) โ€” DeepLearning.AI + Google Cloud. ### T18 ยท Inference, Serving, Cost & Latency +- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**CUDA Mode lectures**](https://github.com/cuda-mode/lectures) โ€” Community lectures on GPU inference internals. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Efficiently Serving LLMs**](https://www.deeplearning.ai/short-courses/efficiently-serving-llms/) โ€” DeepLearning.AI + Predibase. - ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**Quantization Fundamentals with Hugging Face**](https://www.deeplearning.ai/short-courses/quantization-fundamentals-with-hugging-face/) โ€” DeepLearning.AI + HF. -- ๐Ÿง‘โ€๐ŸŽ“ ๐Ÿ†“ [**CUDA Mode lectures**](https://github.com/cuda-mode/lectures) โ€” Community lectures on GPU inference internals. @@ -409,8 +409,8 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T15 ยท Guardrails & Security -- ๐Ÿ“˜ ๐Ÿ’ฐ [**The Developer's Playbook for Large Language Model Security**](https://www.oreilly.com/library/view/the-developers-playbook/9781098162191/) โ€” Steve Wilson (O'Reilly, 2024). OWASP LLM Top 10 project lead's book. - ๐Ÿ“˜ ๐Ÿ’ฐ [**Generative AI Security**](https://link.springer.com/book/10.1007/979-8-8688-0277-1) โ€” Ken Huang et al. (Apress, 2024). +- ๐Ÿ“˜ ๐Ÿ’ฐ [**The Developer's Playbook for Large Language Model Security**](https://www.oreilly.com/library/view/the-developers-playbook/9781098162191/) โ€” Steve Wilson (O'Reilly, 2024). OWASP LLM Top 10 project lead's book. ### T16 ยท Safety, Alignment & Responsible AI @@ -444,150 +444,150 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T1 ยท Coding Agents & AI-Assisted Development -- โญ ๐Ÿ†“ [**Here's how I use LLMs to help me write code**](https://simonwillison.net/2025/Mar/11/using-llms-for-code/) โ€” Simon Willison. - ๐Ÿ†“ [**Agentic Coding: The Future of Software Development**](https://lucumr.pocoo.org/2025/6/12/agentic-coding/) โ€” Armin Ronacher. +- โญ ๐Ÿ†“ [**Here's how I use LLMs to help me write code**](https://simonwillison.net/2025/Mar/11/using-llms-for-code/) โ€” Simon Willison. - ๐Ÿ†“ [**Revenge of the junior developer**](https://sourcegraph.com/blog/revenge-of-the-junior-developer) โ€” Steve Yegge (Sourcegraph). - ๐Ÿ†“ [**The death of the stubborn developer**](https://www.sourcegraph.com/blog/the-death-of-the-junior-developer) โ€” Steve Yegge. ### T2 ยท Spec-Driven Development & Context Engineering -- โญ ๐Ÿ†“ [**The new code**](https://www.latent.space/p/new-code) โ€” Sean Grove / Latent Space. - ๐Ÿ†“ [**Context Engineering**](https://blog.langchain.com/context-engineering-for-agents/) โ€” LangChain. -- ๐Ÿ†“ [**The rise of "context engineering"**](https://blog.langchain.com/the-rise-of-context-engineering/) โ€” LangChain. - ๐Ÿ†“ [**Spec-driven development with AI**](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/) โ€” GitHub Blog. +- โญ ๐Ÿ†“ [**The new code**](https://www.latent.space/p/new-code) โ€” Sean Grove / Latent Space. +- ๐Ÿ†“ [**The rise of "context engineering"**](https://blog.langchain.com/the-rise-of-context-engineering/) โ€” LangChain. ### T3 ยท Agent IDE Rules, Memory Files & Workflows +- ๐Ÿ†“ [**Aider: Tips for using with large codebases**](https://aider.chat/docs/usage/tips.html) โ€” Aider docs. - โญ ๐Ÿ†“ [**Claude Code: Best practices for agentic coding**](https://www.anthropic.com/engineering/claude-code-best-practices) โ€” Anthropic. - ๐Ÿ†“ [**Cursor rules directory**](https://cursor.directory/) โ€” Community catalogue of `.cursorrules` files. - ๐Ÿ†“ [**My Claude Code setup**](https://htdocs.dev/posts/how-to-use-claude-code-to-wield-coding-agent-clusters/) โ€” widely-shared CLAUDE.md + slash-command playbook. -- ๐Ÿ†“ [**Aider: Tips for using with large codebases**](https://aider.chat/docs/usage/tips.html) โ€” Aider docs. ### T4 ยท SWE Benchmarks & Coding Evaluation - โญ ๐Ÿ†“ [**Introducing SWE-bench Verified**](https://openai.com/index/introducing-swe-bench-verified/) โ€” OpenAI. -- ๐Ÿ†“ [**Why we built Terminal-Bench**](https://www.tbench.ai/blog/terminal-bench) โ€” Stanford / Laude. - ๐Ÿ†“ [**Measuring an AI system's ability to do ML R&D**](https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/) โ€” METR. - ๐Ÿ†“ [**The leaderboard illusion**](https://arxiv.org/abs/2504.20879) โ€” Singh et al. on bench-gaming. +- ๐Ÿ†“ [**Why we built Terminal-Bench**](https://www.tbench.ai/blog/terminal-bench) โ€” Stanford / Laude. ### T5 ยท Autonomous Software Agents -- โญ ๐Ÿ†“ [**How we built our multi-agent research system**](https://www.anthropic.com/engineering/built-multi-agent-research-system) โ€” Anthropic. - ๐Ÿ†“ [**Devin, a software engineer**](https://cognition.ai/blog/introducing-devin) โ€” Cognition. - ๐Ÿ†“ [**Don't build multi-agents**](https://cognition.ai/blog/dont-build-multi-agents) โ€” Cognition. Contrarian but important counterpoint to multi-agent maximalism. +- โญ ๐Ÿ†“ [**How we built our multi-agent research system**](https://www.anthropic.com/engineering/built-multi-agent-research-system) โ€” Anthropic. - ๐Ÿ†“ [**SWE-agent: Agent-Computer Interfaces**](https://swe-agent.com/latest/) โ€” Princeton NLP writeup. ### T6 ยท LLM Application Architecture -- โญ ๐Ÿ†“ [**Patterns for Building LLM-based Systems & Products**](https://eugeneyan.com/writing/llm-patterns/) โ€” Eugene Yan. - ๐Ÿ†“ [**Emerging Architectures for LLM Applications**](https://a16z.com/emerging-architectures-for-llm-applications/) โ€” a16z. -- ๐Ÿ†“ [**What We Learned from a Year of Building with LLMs**](https://applied-llms.org/) โ€” Yan/Bensal/Bhawal/Husain/Shankar. +- โญ ๐Ÿ†“ [**Patterns for Building LLM-based Systems & Products**](https://eugeneyan.com/writing/llm-patterns/) โ€” Eugene Yan. - ๐Ÿ†“ [**Twelve factor agents**](https://github.com/humanlayer/12-factor-agents) โ€” HumanLayer. The "12-factor app" equivalent for agent apps. +- ๐Ÿ†“ [**What We Learned from a Year of Building with LLMs**](https://applied-llms.org/) โ€” Yan/Bensal/Bhawal/Husain/Shankar. ### T7 ยท Prompt Engineering +- ๐Ÿ†“ [**A guide to prompting Claude**](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) โ€” Anthropic. - โญ ๐Ÿ†“ [**Prompt Engineering**](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/) โ€” Lilian Weng. - ๐Ÿ†“ [**Prompting is programming**](https://eugeneyan.com/writing/prompting/) โ€” Eugene Yan. -- ๐Ÿ†“ [**A guide to prompting Claude**](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) โ€” Anthropic. - ๐Ÿ†“ [**The prompt report**](https://learnprompting.org/blog/the_prompt_report) โ€” Learn Prompting team summary of their 76-page survey. ### T8 ยท Retrieval-Augmented Generation (RAG) -- โญ ๐Ÿ†“ [**RAG is more than just embedding search**](https://jxnl.co/writing/2024/06/11/rag-is-more-than-just-embedding-search/) โ€” Jason Liu. -- ๐Ÿ†“ [**How to improve your RAG system's performance**](https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1) โ€” Anyscale. - ๐Ÿ†“ [**Advanced RAG Techniques**](https://www.pinecone.io/learn/advanced-rag-techniques/) โ€” Pinecone. +- ๐Ÿ†“ [**How to improve your RAG system's performance**](https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1) โ€” Anyscale. - ๐Ÿ†“ [**Practical considerations in RAG application design**](https://eugeneyan.com/writing/rag/) โ€” Eugene Yan. +- โญ ๐Ÿ†“ [**RAG is more than just embedding search**](https://jxnl.co/writing/2024/06/11/rag-is-more-than-just-embedding-search/) โ€” Jason Liu. ### T9 ยท Memory Systems & Long-Context +- ๐Ÿ†“ [**Extending Context Length in LLMs**](https://huggingface.co/blog/long-range-transformers) โ€” Hugging Face. - โญ ๐Ÿ†“ [**Lost in the Middle: How Language Models Use Long Contexts**](https://arxiv.org/abs/2307.03172) โ€” Liu et al. - ๐Ÿ†“ [**Memory for agents**](https://blog.langchain.com/memory-for-agents/) โ€” LangChain. -- ๐Ÿ†“ [**Extending Context Length in LLMs**](https://huggingface.co/blog/long-range-transformers) โ€” Hugging Face. - ๐Ÿ†“ [**The agentic memory stack**](https://www.letta.com/blog/memgpt) โ€” Letta (MemGPT). ### T10 ยท Tool Use, Function Calling & MCP -- โญ ๐Ÿ†“ [**Introducing the Model Context Protocol**](https://www.anthropic.com/news/model-context-protocol) โ€” Anthropic. +- ๐Ÿ†“ [**Designing MCP servers that agents actually use**](https://www.philschmid.de/mcp-introduction) โ€” Phil Schmid. - ๐Ÿ†“ [**Function calling with LLMs: a practical guide**](https://www.promptingguide.ai/applications/function_calling) โ€” DAIR.AI. +- โญ ๐Ÿ†“ [**Introducing the Model Context Protocol**](https://www.anthropic.com/news/model-context-protocol) โ€” Anthropic. - ๐Ÿ†“ [**Tool use is eating the world**](https://latent.space/p/tools) โ€” Latent Space. -- ๐Ÿ†“ [**Designing MCP servers that agents actually use**](https://www.philschmid.de/mcp-introduction) โ€” Phil Schmid. ### T11 ยท Orchestration & Design Patterns -- โญ ๐Ÿ†“ [**LLM Powered Autonomous Agents**](https://lilianweng.github.io/posts/2023-06-23-agent/) โ€” Lilian Weng. -- ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic. - ๐Ÿ†“ [**Agent design patterns**](https://www.deeplearning.ai/the-batch/issue-241/) โ€” Andrew Ng, The Batch series. - ๐Ÿ†“ [**AI agent frameworks**](https://www.latent.space/p/agent-frameworks) โ€” Latent Space comparative review. +- ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic. +- โญ ๐Ÿ†“ [**LLM Powered Autonomous Agents**](https://lilianweng.github.io/posts/2023-06-23-agent/) โ€” Lilian Weng. ### T12 ยท Multi-Agent Systems & Coordination +- ๐Ÿ†“ [**AutoGen: Enabling next-gen LLM applications**](https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/) โ€” Microsoft. +- ๐Ÿ†“ [**Don't build multi-agents**](https://cognition.ai/blog/dont-build-multi-agents) โ€” Cognition. - โญ ๐Ÿ†“ [**How we built our multi-agent research system**](https://www.anthropic.com/engineering/built-multi-agent-research-system) โ€” Anthropic. - ๐Ÿ†“ [**Multi-agent workflows**](https://blog.langchain.com/langgraph-multi-agent-workflows/) โ€” LangChain. -- ๐Ÿ†“ [**Don't build multi-agents**](https://cognition.ai/blog/dont-build-multi-agents) โ€” Cognition. -- ๐Ÿ†“ [**AutoGen: Enabling next-gen LLM applications**](https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/) โ€” Microsoft. ### T13 ยท Evaluation & Testing -- โญ ๐Ÿ†“ [**Your AI product needs evals**](https://hamel.dev/blog/posts/evals/) โ€” Hamel Husain. -- ๐Ÿ†“ [**Task-specific LLM evals that do & don't work**](https://eugeneyan.com/writing/evals/) โ€” Eugene Yan. - ๐Ÿ†“ [**Creating a LLM-as-a-Judge that drives business results**](https://hamel.dev/blog/posts/llm-judge/) โ€” Hamel Husain. - ๐Ÿ†“ [**LLM evals: everything I learned in 12 months**](https://www.shreya-shankar.com/evals-paper/) โ€” Shreya Shankar. +- ๐Ÿ†“ [**Task-specific LLM evals that do & don't work**](https://eugeneyan.com/writing/evals/) โ€” Eugene Yan. +- โญ ๐Ÿ†“ [**Your AI product needs evals**](https://hamel.dev/blog/posts/evals/) โ€” Hamel Husain. ### T14 ยท Observability, Tracing & Debugging -- โญ ๐Ÿ†“ [**So you want to build an LLM observability platform**](https://hamel.dev/blog/posts/evals/#observability) โ€” Hamel Husain (subsection of evals post; foundational). -- ๐Ÿ†“ [**The OpenTelemetry Gen AI semantic conventions**](https://opentelemetry.io/docs/specs/semconv/gen-ai/) โ€” OTel. - ๐Ÿ†“ [**How Honeycomb uses LLMs for product experiences**](https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm) โ€” Phillip Carter. - ๐Ÿ†“ [**Logfire: observability for the LLM era**](https://pydantic.dev/logfire) โ€” Pydantic. +- โญ ๐Ÿ†“ [**So you want to build an LLM observability platform**](https://hamel.dev/blog/posts/evals/#observability) โ€” Hamel Husain (subsection of evals post; foundational). +- ๐Ÿ†“ [**The OpenTelemetry Gen AI semantic conventions**](https://opentelemetry.io/docs/specs/semconv/gen-ai/) โ€” OTel. ### T15 ยท Guardrails & Security -- โญ ๐Ÿ†“ [**Prompt injection series**](https://simonwillison.net/series/prompt-injection/) โ€” Simon Willison. Canonical ongoing series. - ๐Ÿ†“ [**OWASP Top 10 for LLM Applications**](https://genai.owasp.org/llm-top-10/) โ€” OWASP. -- ๐Ÿ†“ [**Universal and Transferable Adversarial Attacks on Aligned LLMs**](https://arxiv.org/abs/2307.15043) โ€” Zou et al. (GCG attack). +- โญ ๐Ÿ†“ [**Prompt injection series**](https://simonwillison.net/series/prompt-injection/) โ€” Simon Willison. Canonical ongoing series. - ๐Ÿ†“ [**Red teaming LLMs**](https://huggingface.co/blog/red-teaming) โ€” Hugging Face. +- ๐Ÿ†“ [**Universal and Transferable Adversarial Attacks on Aligned LLMs**](https://arxiv.org/abs/2307.15043) โ€” Zou et al. (GCG attack). ### T16 ยท Safety, Alignment & Responsible AI -- โญ ๐Ÿ†“ [**Core Views on AI Safety**](https://www.anthropic.com/news/core-views-on-ai-safety) โ€” Anthropic. - ๐Ÿ†“ [**Anthropic's Responsible Scaling Policy**](https://www.anthropic.com/news/anthropics-responsible-scaling-policy) โ€” Anthropic. +- โญ ๐Ÿ†“ [**Core Views on AI Safety**](https://www.anthropic.com/news/core-views-on-ai-safety) โ€” Anthropic. - ๐Ÿ†“ [**Preparedness Framework**](https://openai.com/safety/preparedness) โ€” OpenAI. - ๐Ÿ†“ [**Scalable oversight via debate & recursive reward modelling**](https://deepmindsafetyresearch.medium.com/) โ€” DeepMind Safety Research. ### T17 ยท Fine-tuning, Post-training & RLHF - โญ ๐Ÿ†“ [**Ahead of AI**](https://magazine.sebastianraschka.com/) โ€” Sebastian Raschka. The canonical fine-tuning / post-training deep-dives. -- ๐Ÿ†“ [**The Novice's LLM Training Guide**](https://rentry.org/llm-training) โ€” Community reference. - ๐Ÿ†“ [**DPO: Your language model is secretly a reward model**](https://arxiv.org/abs/2305.18290) โ€” Rafailov et al. - ๐Ÿ†“ [**The alignment handbook**](https://github.com/huggingface/alignment-handbook) โ€” Hugging Face. +- ๐Ÿ†“ [**The Novice's LLM Training Guide**](https://rentry.org/llm-training) โ€” Community reference. ### T18 ยท Inference, Serving, Cost & Latency -- โญ ๐Ÿ†“ [**Transformer Inference Arithmetic**](https://kipp.ly/transformer-inference-arithmetic/) โ€” Kipply. -- ๐Ÿ†“ [**LLM Inference Speed of Light**](https://zeux.io/2024/03/15/llm-inference-sol/) โ€” Arseny Kapoulkine. - ๐Ÿ†“ [**Everything I've learned about efficient LLM inference**](https://www.baseten.co/blog/) โ€” Baseten engineering blog. - ๐Ÿ†“ [**GPU performance for LLM inference**](https://blog.vllm.ai/) โ€” vLLM team blog. +- ๐Ÿ†“ [**LLM Inference Speed of Light**](https://zeux.io/2024/03/15/llm-inference-sol/) โ€” Arseny Kapoulkine. +- โญ ๐Ÿ†“ [**Transformer Inference Arithmetic**](https://kipp.ly/transformer-inference-arithmetic/) โ€” Kipply. ### T19 ยท Voice, Multi-modal & Embodied Agents -- โญ ๐Ÿ†“ [**Hello GPT-4o**](https://openai.com/index/hello-gpt-4o/) โ€” OpenAI. - ๐Ÿ†“ [**Building a voice agent with LiveKit**](https://docs.livekit.io/agents/) โ€” LiveKit Agents docs. -- ๐Ÿ†“ [**Voice-first LLM products**](https://www.latent.space/p/voice-2024) โ€” Latent Space. +- โญ ๐Ÿ†“ [**Hello GPT-4o**](https://openai.com/index/hello-gpt-4o/) โ€” OpenAI. - ๐Ÿ†“ [**Moshi: a speech-text foundation model**](https://kyutai.org/Moshi.pdf) โ€” Kyutai. +- ๐Ÿ†“ [**Voice-first LLM products**](https://www.latent.space/p/voice-2024) โ€” Latent Space. ### T20 ยท Product, UX & Human-AI Interaction +- ๐Ÿ†“ [**Building products with AI: UX lessons**](https://www.linusakesson.net/) / [**thesephist.com essays**](https://thesephist.com/) โ€” Linus Lee. +- ๐Ÿ†“ [**Generative AI: Design Patterns (NNGroup)**](https://www.nngroup.com/articles/generative-ai-design-patterns/) โ€” Nielsen Norman Group. - โญ ๐Ÿ†“ [**Maggie Appleton essays**](https://maggieappleton.com/) โ€” Canonical AI-UX thinking. - ๐Ÿ†“ [**Microsoft HAX guidelines for human-AI interaction**](https://www.microsoft.com/en-us/research/project/guidelines-for-human-ai-interaction/) โ€” Microsoft Research. -- ๐Ÿ†“ [**Generative AI: Design Patterns (NNGroup)**](https://www.nngroup.com/articles/generative-ai-design-patterns/) โ€” Nielsen Norman Group. -- ๐Ÿ†“ [**Building products with AI: UX lessons**](https://www.linusakesson.net/) / [**thesephist.com essays**](https://thesephist.com/) โ€” Linus Lee. ### T21 ยท Economics, Teams, Hiring & Org Design +- ๐Ÿ†“ [**16 Changes to the Way Enterprises Build Software with AI**](https://a16z.com/16-changes-to-the-way-enterprises-are-building-and-buying-generative-ai/) โ€” a16z. +- ๐Ÿ†“ [**a16z AI canon**](https://a16z.com/ai-canon/) โ€” a16z. - โญ ๐Ÿ†“ [**AI engineering org design**](https://newsletter.pragmaticengineer.com/t/ai) โ€” Gergely Orosz, Pragmatic Engineer. - ๐Ÿ†“ [**Building an AI team**](https://eugeneyan.com/writing/team-size/) โ€” Eugene Yan. -- ๐Ÿ†“ [**a16z AI canon**](https://a16z.com/ai-canon/) โ€” a16z. -- ๐Ÿ†“ [**16 Changes to the Way Enterprises Build Software with AI**](https://a16z.com/16-changes-to-the-way-enterprises-are-building-and-buying-generative-ai/) โ€” a16z. @@ -597,14 +597,14 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T1 ยท Coding Agents & AI-Assisted Development -- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**Claude Code cookbook**](https://github.com/anthropics/claude-code) โ€” Anthropic. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Aider tutorials**](https://aider.chat/docs/usage/tutorials.html) โ€” Aider docs. +- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**Claude Code cookbook**](https://github.com/anthropics/claude-code) โ€” Anthropic. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Continue.dev recipes**](https://docs.continue.dev/customize/tutorials/) โ€” Continue. ### T2 ยท Spec-Driven Development -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**GitHub spec-kit**](https://github.com/github/spec-kit) โ€” The official spec-driven-development toolkit. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**AGENTS.md examples**](https://agents.md/examples) โ€” Example `AGENTS.md` files for common stacks. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**GitHub spec-kit**](https://github.com/github/spec-kit) โ€” The official spec-driven-development toolkit. ### T3 ยท Agent IDE Rules & Workflows @@ -613,15 +613,15 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T5 ยท Autonomous Software Agents -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**SWE-agent quickstart**](https://swe-agent.com/latest/usage/coding_challenges/) โ€” Princeton NLP. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**OpenHands (formerly OpenDevin)**](https://docs.all-hands.dev/) โ€” All Hands AI. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**SWE-agent quickstart**](https://swe-agent.com/latest/usage/coding_challenges/) โ€” Princeton NLP. ### T6 ยท LLM Application Architecture -- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**OpenAI Cookbook**](https://cookbook.openai.com/) โ€” The reference recipe library for OpenAI APIs. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Anthropic Cookbook**](https://github.com/anthropics/anthropic-cookbook) โ€” Claude recipes. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Gemini API Cookbook**](https://github.com/google-gemini/cookbook) โ€” Google. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Hugging Face Open-Source AI Cookbook**](https://huggingface.co/learn/cookbook/) โ€” Hugging Face. +- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**OpenAI Cookbook**](https://cookbook.openai.com/) โ€” The reference recipe library for OpenAI APIs. ### T7 ยท Prompt Engineering @@ -630,33 +630,33 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T8 ยท Retrieval-Augmented Generation (RAG) -- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**LlamaIndex tutorials**](https://docs.llamaindex.ai/en/stable/getting_started/starter_example/) โ€” LlamaIndex. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Advanced RAG notebooks**](https://github.com/NirDiamant/RAG_Techniques) โ€” Nir Diamant. 30+ advanced RAG recipes. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**LangChain RAG from scratch**](https://github.com/langchain-ai/rag-from-scratch) โ€” LangChain. +- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**LlamaIndex tutorials**](https://docs.llamaindex.ai/en/stable/getting_started/starter_example/) โ€” LlamaIndex. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Pinecone RAG handbook**](https://www.pinecone.io/learn/retrieval-augmented-generation/) โ€” Pinecone. -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Advanced RAG notebooks**](https://github.com/NirDiamant/RAG_Techniques) โ€” Nir Diamant. 30+ advanced RAG recipes. ### T9 ยท Memory Systems -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Mem0 quickstart**](https://docs.mem0.ai/) โ€” Mem0. -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Letta (MemGPT) cookbook**](https://docs.letta.com/) โ€” Letta. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**LangGraph memory**](https://langchain-ai.github.io/langgraph/concepts/memory/) โ€” LangChain. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Letta (MemGPT) cookbook**](https://docs.letta.com/) โ€” Letta. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Mem0 quickstart**](https://docs.mem0.ai/) โ€” Mem0. ### T10 ยท Tool Use & MCP -- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**MCP quickstart**](https://modelcontextprotocol.io/quickstart) โ€” Anthropic. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**awesome-mcp-servers**](https://github.com/punkpeye/awesome-mcp-servers) โ€” Community reference-servers catalogue. +- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**MCP quickstart**](https://modelcontextprotocol.io/quickstart) โ€” Anthropic. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**OpenAI function calling cookbook**](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models) โ€” OpenAI. ### T11 ยท Orchestration & Patterns -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**LangGraph tutorials**](https://langchain-ai.github.io/langgraph/tutorials/) โ€” LangChain. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Anthropic building-effective-agents examples**](https://github.com/anthropics/anthropic-cookbook/tree/main/patterns/agents) โ€” Anthropic. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**LangGraph tutorials**](https://langchain-ai.github.io/langgraph/tutorials/) โ€” LangChain. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**LlamaIndex agent tutorials**](https://docs.llamaindex.ai/en/stable/module_guides/deploying/agents/) โ€” LlamaIndex. ### T12 ยท Multi-Agent Systems -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**CrewAI examples**](https://github.com/crewAIInc/crewAI-examples) โ€” CrewAI. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**AutoGen notebook gallery**](https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/tutorial/introduction.html) โ€” Microsoft. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**CrewAI examples**](https://github.com/crewAIInc/crewAI-examples) โ€” CrewAI. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**LangGraph multi-agent examples**](https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/) โ€” LangChain. ### T13 ยท Evaluation & Testing @@ -667,8 +667,8 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T14 ยท Observability -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Langfuse cookbook**](https://langfuse.com/docs/integrations/overview) โ€” Langfuse. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Arize Phoenix tutorials**](https://docs.arize.com/phoenix/tutorials) โ€” Arize. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**Langfuse cookbook**](https://langfuse.com/docs/integrations/overview) โ€” Langfuse. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Logfire LLM tracing tutorials**](https://logfire.pydantic.dev/docs/guides/onboarding-checklist/integrate/) โ€” Pydantic. ### T15 ยท Guardrails & Security @@ -679,20 +679,20 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T17 ยท Fine-tuning & Post-training -- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**Unsloth notebooks**](https://github.com/unslothai/unsloth) โ€” Fast fine-tuning recipes. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Axolotl examples**](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples) โ€” Axolotl. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Hugging Face TRL tutorials**](https://huggingface.co/docs/trl/) โ€” TRL. +- โญ ๐Ÿ› ๏ธ ๐Ÿ†“ [**Unsloth notebooks**](https://github.com/unslothai/unsloth) โ€” Fast fine-tuning recipes. ### T18 ยท Inference & Serving -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**vLLM examples**](https://docs.vllm.ai/en/latest/getting_started/examples/examples_index.html) โ€” vLLM. -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**TensorRT-LLM tutorials**](https://nvidia.github.io/TensorRT-LLM/) โ€” NVIDIA. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**llama.cpp server**](https://github.com/ggerganov/llama.cpp/tree/master/examples/server) โ€” ggerganov. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**TensorRT-LLM tutorials**](https://nvidia.github.io/TensorRT-LLM/) โ€” NVIDIA. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**vLLM examples**](https://docs.vllm.ai/en/latest/getting_started/examples/examples_index.html) โ€” vLLM. ### T19 ยท Voice & Multimodal -- ๐Ÿ› ๏ธ ๐Ÿ†“ [**OpenAI Realtime API cookbook**](https://cookbook.openai.com/examples/realtime_api_examples) โ€” OpenAI. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**LiveKit Agents examples**](https://github.com/livekit/agents) โ€” LiveKit. +- ๐Ÿ› ๏ธ ๐Ÿ†“ [**OpenAI Realtime API cookbook**](https://cookbook.openai.com/examples/realtime_api_examples) โ€” OpenAI. - ๐Ÿ› ๏ธ ๐Ÿ†“ [**Pipecat**](https://github.com/pipecat-ai/pipecat) โ€” Daily. Voice-agent framework with extensive cookbook. @@ -701,26 +701,26 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR *Opinionated, prescriptive guides distilling design patterns and operational practices.* -- โญ ๐Ÿ“‹ ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic. The canonical pattern taxonomy (T11). -- โญ ๐Ÿ“‹ ๐Ÿ†“ [**Patterns for Building LLM-based Systems & Products**](https://eugeneyan.com/writing/llm-patterns/) โ€” Eugene Yan (T6). -- โญ ๐Ÿ“‹ ๐Ÿ†“ [**What We Learned from a Year of Building with LLMs**](https://applied-llms.org/) โ€” Yan/Bensal/Bhawal/Husain/Shankar (T6/T13). - ๐Ÿ“‹ ๐Ÿ†“ [**12-Factor Agents**](https://github.com/humanlayer/12-factor-agents) โ€” HumanLayer. Opinionated operational principles for agent apps (T6/T11). - ๐Ÿ“‹ ๐Ÿ†“ [**A practical guide to building agents**](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf) โ€” OpenAI PDF (T11). +- ๐Ÿ“‹ ๐Ÿ†“ [**a16z AI canon**](https://a16z.com/ai-canon/) โ€” a16z (T20/T21). +- ๐Ÿ“‹ ๐Ÿ†“ [**Agentic UX**](https://agentic-ux.com/) โ€” 11 runtime lifecycle patterns for supervised delegation, organized before/while/after an agent acts, with interactive mockups, production screenshots, and an MCP server for coding agents (Daniel Albinsson, 2025). +- ๐Ÿ“‹ ๐Ÿ†“ [**Anthropic's prompt engineering overview**](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) โ€” Anthropic (T7). +- โญ ๐Ÿ“‹ ๐Ÿ†“ [**Building effective agents**](https://www.anthropic.com/engineering/building-effective-agents) โ€” Anthropic. The canonical pattern taxonomy (T11). - ๐Ÿ“‹ ๐Ÿ†“ [**Claude Code: best practices for agentic coding**](https://www.anthropic.com/engineering/claude-code-best-practices) โ€” Anthropic (T1/T3). -- ๐Ÿ“‹ ๐Ÿ†“ [**LangGraph design patterns**](https://langchain-ai.github.io/langgraph/concepts/) โ€” LangChain (T11/T12). - ๐Ÿ“‹ ๐Ÿ†“ [**Instructor's RAG patterns**](https://python.useinstructor.com/blog/2024/10/23/systematically-improving-your-rag/) โ€” Jason Liu (T8). -- ๐Ÿ“‹ ๐Ÿ†“ [**OpenAI's prompt-engineering playbook**](https://platform.openai.com/docs/guides/prompt-engineering) โ€” OpenAI (T7). -- ๐Ÿ“‹ ๐Ÿ†“ [**Anthropic's prompt engineering overview**](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview) โ€” Anthropic (T7). -- ๐Ÿ“‹ ๐Ÿ†“ [**RAG-Fusion, HyDE, and other advanced retrieval patterns**](https://github.com/NirDiamant/RAG_Techniques) โ€” Nir Diamant (T8). +- ๐Ÿ“‹ ๐Ÿ†“ [**LangGraph design patterns**](https://langchain-ai.github.io/langgraph/concepts/) โ€” LangChain (T11/T12). - ๐Ÿ“‹ ๐Ÿ†“ [**LLM observability playbook**](https://hamel.dev/blog/posts/evals/) โ€” Hamel Husain (T13/T14). -- ๐Ÿ“‹ ๐Ÿ†“ [**OWASP Top 10 for LLM Applications**](https://genai.owasp.org/llm-top-10/) โ€” OWASP. The security-pattern catalogue (T15). - ๐Ÿ“‹ ๐Ÿ†“ [**MITRE ATLAS**](https://atlas.mitre.org/) โ€” Adversarial Threat Landscape for AI Systems (T15). - ๐Ÿ“‹ ๐Ÿ†“ [**NIST AI Risk Management Framework**](https://www.nist.gov/itl/ai-risk-management-framework) โ€” NIST (T16). -- ๐Ÿ“‹ ๐Ÿ†“ [**The LLM inference playbook**](https://github.com/ray-project/llm-applications) โ€” Anyscale (T18). +- ๐Ÿ“‹ ๐Ÿ†“ [**OpenAI's prompt-engineering playbook**](https://platform.openai.com/docs/guides/prompt-engineering) โ€” OpenAI (T7). +- ๐Ÿ“‹ ๐Ÿ†“ [**OWASP Top 10 for LLM Applications**](https://genai.owasp.org/llm-top-10/) โ€” OWASP. The security-pattern catalogue (T15). +- โญ ๐Ÿ“‹ ๐Ÿ†“ [**Patterns for Building LLM-based Systems & Products**](https://eugeneyan.com/writing/llm-patterns/) โ€” Eugene Yan (T6). - ๐Ÿ“‹ ๐Ÿ†“ [**Prompt-injection defence patterns**](https://simonwillison.net/2023/Apr/14/worst-that-can-happen/) โ€” Simon Willison (T15). -- ๐Ÿ“‹ ๐Ÿ†“ [**a16z AI canon**](https://a16z.com/ai-canon/) โ€” a16z (T20/T21). -- ๐Ÿ“‹ ๐Ÿ†“ [**Agentic UX**](https://agentic-ux.com/) โ€” 11 runtime lifecycle patterns for supervised delegation, organized before/while/after an agent acts, with interactive mockups, production screenshots, and an MCP server for coding agents (Daniel Albinsson, 2025). +- ๐Ÿ“‹ ๐Ÿ†“ [**RAG-Fusion, HyDE, and other advanced retrieval patterns**](https://github.com/NirDiamant/RAG_Techniques) โ€” Nir Diamant (T8). +- ๐Ÿ“‹ ๐Ÿ†“ [**The LLM inference playbook**](https://github.com/ray-project/llm-applications) โ€” Anyscale (T18). - ๐Ÿ“‹ ๐Ÿ†“ [**UX design patterns for AI products**](https://www.nngroup.com/articles/generative-ai-design-patterns/) โ€” Nielsen Norman Group (T20). +- โญ ๐Ÿ“‹ ๐Ÿ†“ [**What We Learned from a Year of Building with LLMs**](https://applied-llms.org/) โ€” Yan/Bensal/Bhawal/Husain/Shankar (T6/T13). @@ -765,17 +765,17 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T1 ยท Coding Agents & T4 ยท SWE Benchmarks -- ๐Ÿ“„ ๐Ÿ†“ [**SWE-bench: Can LMs Resolve Real-World GitHub Issues?**](https://arxiv.org/abs/2310.06770) โ€” Jimenez et al. -- ๐Ÿ“„ ๐Ÿ†“ [**SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering**](https://arxiv.org/abs/2405.15793) โ€” Yang et al. - ๐Ÿ“„ ๐Ÿ†“ [**AutoCodeRover: Autonomous Program Improvement**](https://arxiv.org/abs/2404.05427) โ€” Zhang et al. -- ๐Ÿ“„ ๐Ÿ†“ [**LiveCodeBench**](https://arxiv.org/abs/2403.07974) โ€” Jain et al. - ๐Ÿ“„ ๐Ÿ†“ [**BigCodeBench**](https://arxiv.org/abs/2406.15877) โ€” Zhuo et al. +- ๐Ÿ“„ ๐Ÿ†“ [**LiveCodeBench**](https://arxiv.org/abs/2403.07974) โ€” Jain et al. +- ๐Ÿ“„ ๐Ÿ†“ [**SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering**](https://arxiv.org/abs/2405.15793) โ€” Yang et al. +- ๐Ÿ“„ ๐Ÿ†“ [**SWE-bench: Can LMs Resolve Real-World GitHub Issues?**](https://arxiv.org/abs/2310.06770) โ€” Jimenez et al. ### T5 ยท Autonomous SWE Agents -- ๐Ÿ“„ ๐Ÿ†“ [**Voyager: An Open-Ended Embodied Agent with LLMs**](https://arxiv.org/abs/2305.16291) โ€” Wang et al. - ๐Ÿ“„ ๐Ÿ†“ [**Agentless: Demystifying LLM-based Software Engineering Agents**](https://arxiv.org/abs/2407.01489) โ€” Xia et al. - ๐Ÿ“„ ๐Ÿ†“ [**OpenHands / OpenDevin**](https://arxiv.org/abs/2407.16741) โ€” All Hands AI. +- ๐Ÿ“„ ๐Ÿ†“ [**Voyager: An Open-Ended Embodied Agent with LLMs**](https://arxiv.org/abs/2305.16291) โ€” Wang et al. ### T6 ยท App Architecture @@ -785,30 +785,30 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T7 ยท Prompt Engineering - ๐Ÿ“„ ๐Ÿ†“ [**Chain-of-Thought Prompting Elicits Reasoning**](https://arxiv.org/abs/2201.11903) โ€” Wei et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Tree of Thoughts**](https://arxiv.org/abs/2305.10601) โ€” Yao et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Self-Consistency Improves CoT**](https://arxiv.org/abs/2203.11171) โ€” Wang et al. - ๐Ÿ“„ ๐Ÿ†“ [**Large Language Models are Zero-Shot Reasoners**](https://arxiv.org/abs/2205.11916) โ€” Kojima et al. ("Let's think step by step"). +- ๐Ÿ“„ ๐Ÿ†“ [**Self-Consistency Improves CoT**](https://arxiv.org/abs/2203.11171) โ€” Wang et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Tree of Thoughts**](https://arxiv.org/abs/2305.10601) โ€” Yao et al. ### T8 ยท RAG -- ๐Ÿ“„ ๐Ÿ†“ [**Retrieval-Augmented Generation for Knowledge-Intensive NLP**](https://arxiv.org/abs/2005.11401) โ€” Lewis et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Dense Passage Retrieval**](https://arxiv.org/abs/2004.04906) โ€” Karpukhin et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE)**](https://arxiv.org/abs/2212.10496) โ€” Gao et al. - ๐Ÿ“„ ๐Ÿ†“ [**RAG for LLMs: A Survey**](https://arxiv.org/abs/2312.10997) โ€” Gao et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Retrieval-Augmented Generation for Knowledge-Intensive NLP**](https://arxiv.org/abs/2005.11401) โ€” Lewis et al. - ๐Ÿ“„ ๐Ÿ†“ [**Self-RAG: Learning to Retrieve, Generate, and Critique**](https://arxiv.org/abs/2310.11511) โ€” Asai et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE)**](https://arxiv.org/abs/2212.10496) โ€” Gao et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Dense Passage Retrieval**](https://arxiv.org/abs/2004.04906) โ€” Karpukhin et al. ### T9 ยท Memory -- ๐Ÿ“„ ๐Ÿ†“ [**MemGPT: Towards LLMs as Operating Systems**](https://arxiv.org/abs/2310.08560) โ€” Packer et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Lost in the Middle**](https://arxiv.org/abs/2307.03172) โ€” Liu et al. - ๐Ÿ“„ ๐Ÿ†“ [**Generative Agents: Interactive Simulacra of Human Behavior**](https://arxiv.org/abs/2304.03442) โ€” Park et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Lost in the Middle**](https://arxiv.org/abs/2307.03172) โ€” Liu et al. +- ๐Ÿ“„ ๐Ÿ†“ [**MemGPT: Towards LLMs as Operating Systems**](https://arxiv.org/abs/2310.08560) โ€” Packer et al. ### T10 ยท Tool Use & MCP -- ๐Ÿ“„ ๐Ÿ†“ [**Toolformer**](https://arxiv.org/abs/2302.04761) โ€” Schick et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Berkeley Function-Calling Leaderboard**](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) โ€” UC Berkeley. - ๐Ÿ“„ ๐Ÿ†“ [**Gorilla: LLM Connected with Massive APIs**](https://arxiv.org/abs/2305.15334) โ€” Patil et al. - ๐Ÿ“„ ๐Ÿ†“ [**MRKL Systems**](https://arxiv.org/abs/2205.00445) โ€” Karpas et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Berkeley Function-Calling Leaderboard**](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) โ€” UC Berkeley. +- ๐Ÿ“„ ๐Ÿ†“ [**Toolformer**](https://arxiv.org/abs/2302.04761) โ€” Schick et al. ### T11 ยท Orchestration & Patterns @@ -819,15 +819,15 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T12 ยท Multi-Agent +- ๐Ÿ“„ ๐Ÿ†“ [**A Survey on LLM-based Autonomous Agents**](https://arxiv.org/abs/2308.11432) โ€” Wang et al. - ๐Ÿ“„ ๐Ÿ†“ [**AutoGen**](https://arxiv.org/abs/2308.08155) โ€” Wu et al. - ๐Ÿ“„ ๐Ÿ†“ [**CAMEL: Communicative Agents for Mind Exploration**](https://arxiv.org/abs/2303.17760) โ€” Li et al. -- ๐Ÿ“„ ๐Ÿ†“ [**A Survey on LLM-based Autonomous Agents**](https://arxiv.org/abs/2308.11432) โ€” Wang et al. - ๐Ÿ“„ ๐Ÿ†“ [**MetaGPT**](https://arxiv.org/abs/2308.00352) โ€” Hong et al. ### T13 ยท Evaluation -- ๐Ÿ“„ ๐Ÿ†“ [**Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena**](https://arxiv.org/abs/2306.05685) โ€” Zheng et al. - ๐Ÿ“„ ๐Ÿ†“ [**HELM: Holistic Evaluation of Language Models**](https://arxiv.org/abs/2211.09110) โ€” Liang et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena**](https://arxiv.org/abs/2306.05685) โ€” Zheng et al. - ๐Ÿ“„ ๐Ÿ†“ [**Who Validates the Validators?**](https://arxiv.org/abs/2404.12272) โ€” Shankar et al. ### T14 ยท Observability @@ -836,35 +836,35 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T15 ยท Guardrails & Security -- ๐Ÿ“„ ๐Ÿ†“ [**Universal and Transferable Adversarial Attacks on Aligned LLMs (GCG)**](https://arxiv.org/abs/2307.15043) โ€” Zou et al. - ๐Ÿ“„ ๐Ÿ†“ [**Many-shot Jailbreaking**](https://www.anthropic.com/research/many-shot-jailbreaking) โ€” Anthropic. - ๐Ÿ“„ ๐Ÿ†“ [**Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection**](https://arxiv.org/abs/2302.12173) โ€” Greshake et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Universal and Transferable Adversarial Attacks on Aligned LLMs (GCG)**](https://arxiv.org/abs/2307.15043) โ€” Zou et al. ### T16 ยท Safety & Alignment +- ๐Ÿ“„ ๐Ÿ†“ [**Concrete Problems in AI Safety**](https://arxiv.org/abs/1606.06565) โ€” Amodei et al. - ๐Ÿ“„ ๐Ÿ†“ [**Constitutional AI**](https://arxiv.org/abs/2212.08073) โ€” Bai et al. - ๐Ÿ“„ ๐Ÿ†“ [**Scalable Agent Alignment via Reward Modeling**](https://arxiv.org/abs/1811.07871) โ€” Leike et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Concrete Problems in AI Safety**](https://arxiv.org/abs/1606.06565) โ€” Amodei et al. ### T17 ยท Fine-tuning & Post-training +- ๐Ÿ“„ ๐Ÿ†“ [**Constitutional AI / RLAIF**](https://arxiv.org/abs/2212.08073) โ€” Bai et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Direct Preference Optimization (DPO)**](https://arxiv.org/abs/2305.18290) โ€” Rafailov et al. - ๐Ÿ“„ ๐Ÿ†“ [**LoRA: Low-Rank Adaptation**](https://arxiv.org/abs/2106.09685) โ€” Hu et al. - ๐Ÿ“„ ๐Ÿ†“ [**QLoRA**](https://arxiv.org/abs/2305.14314) โ€” Dettmers et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Direct Preference Optimization (DPO)**](https://arxiv.org/abs/2305.18290) โ€” Rafailov et al. - ๐Ÿ“„ ๐Ÿ†“ [**Training LMs to follow instructions with human feedback (InstructGPT)**](https://arxiv.org/abs/2203.02155) โ€” Ouyang et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Constitutional AI / RLAIF**](https://arxiv.org/abs/2212.08073) โ€” Bai et al. ### T18 ยท Inference & Serving - ๐Ÿ“„ ๐Ÿ†“ [**Efficient Memory Management for LLM Serving with PagedAttention (vLLM)**](https://arxiv.org/abs/2309.06180) โ€” Kwon et al. +- ๐Ÿ“„ ๐Ÿ†“ [**Fast Inference from Transformers via Speculative Decoding**](https://arxiv.org/abs/2211.17192) โ€” Leviathan et al. - ๐Ÿ“„ ๐Ÿ†“ [**FlashAttention**](https://arxiv.org/abs/2205.14135) โ€” Dao et al. - ๐Ÿ“„ ๐Ÿ†“ [**SGLang: Efficient Execution of Structured Language Model Programs**](https://arxiv.org/abs/2312.07104) โ€” Zheng et al. -- ๐Ÿ“„ ๐Ÿ†“ [**Fast Inference from Transformers via Speculative Decoding**](https://arxiv.org/abs/2211.17192) โ€” Leviathan et al. ### T19 ยท Voice & Multimodal -- ๐Ÿ“„ ๐Ÿ†“ [**Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)**](https://arxiv.org/abs/2212.04356) โ€” Radford et al. - ๐Ÿ“„ ๐Ÿ†“ [**Moshi**](https://kyutai.org/Moshi.pdf) โ€” Kyutai. +- ๐Ÿ“„ ๐Ÿ†“ [**Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)**](https://arxiv.org/abs/2212.04356) โ€” Radford et al. - ๐Ÿ“„ ๐Ÿ†“ [**Seamless: Multilingual Expressive and Streaming Speech Translation**](https://arxiv.org/abs/2312.05187) โ€” Meta. ### T20 ยท Product & UX @@ -879,33 +879,33 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T1 / T4 ยท Coding Agents & SWE Benchmarks -- โญ ๐Ÿงช ๐Ÿ†“ [**SWE-bench**](https://www.swebench.com/) โ€” Real-world GitHub-issue resolution benchmark; Verified subset is the de-facto industry standard. -- ๐Ÿงช ๐Ÿ†“ [**Terminal-Bench**](https://www.tbench.ai/) โ€” Stanford / Laude. Long-horizon terminal task benchmark. -- ๐Ÿงช ๐Ÿ†“ [**LiveCodeBench**](https://livecodebench.github.io/) โ€” Rolling contamination-free coding benchmark. - ๐Ÿงช ๐Ÿ†“ [**BigCodeBench**](https://bigcode-bench.github.io/) โ€” Practical programming with diverse function calls. - ๐Ÿงช ๐Ÿ†“ [**HumanEval+ / EvalPlus**](https://evalplus.github.io/) โ€” Strengthened HumanEval. +- ๐Ÿงช ๐Ÿ†“ [**LiveCodeBench**](https://livecodebench.github.io/) โ€” Rolling contamination-free coding benchmark. - ๐Ÿงช ๐Ÿ†“ [**MLE-bench**](https://github.com/openai/mle-bench) โ€” OpenAI. Kaggle-style ML engineering benchmark. +- โญ ๐Ÿงช ๐Ÿ†“ [**SWE-bench**](https://www.swebench.com/) โ€” Real-world GitHub-issue resolution benchmark; Verified subset is the de-facto industry standard. +- ๐Ÿงช ๐Ÿ†“ [**Terminal-Bench**](https://www.tbench.ai/) โ€” Stanford / Laude. Long-horizon terminal task benchmark. ### T5 ยท Autonomous Agents -- ๐Ÿงช ๐Ÿ†“ [**GAIA**](https://huggingface.co/gaia-benchmark) โ€” General AI Assistants benchmark. - ๐Ÿงช ๐Ÿ†“ [**AgentBench**](https://llmbench.ai/agent) โ€” Tsinghua. Broad agent capability benchmark. -- ๐Ÿงช ๐Ÿ†“ [**WebArena**](https://webarena.dev/) / [**VisualWebArena**](https://jykoh.com/vwa) โ€” Web-navigation agents. -- ๐Ÿงช ๐Ÿ†“ [**OSWorld**](https://os-world.github.io/) โ€” Desktop OS-controlling agents. +- ๐Ÿงช ๐Ÿ†“ [**GAIA**](https://huggingface.co/gaia-benchmark) โ€” General AI Assistants benchmark. - ๐Ÿงช ๐Ÿ†“ [**MLE-bench**](https://github.com/openai/mle-bench) โ€” ML-engineering agents. +- ๐Ÿงช ๐Ÿ†“ [**OSWorld**](https://os-world.github.io/) โ€” Desktop OS-controlling agents. +- ๐Ÿงช ๐Ÿ†“ [**WebArena**](https://webarena.dev/) / [**VisualWebArena**](https://jykoh.com/vwa) โ€” Web-navigation agents. ### T8 ยท RAG -- ๐Ÿงช ๐Ÿ†“ [**RAGAS**](https://docs.ragas.io/) โ€” Framework and leaderboard for RAG eval. -- ๐Ÿงช ๐Ÿ†“ [**MTEB**](https://huggingface.co/spaces/mteb/leaderboard) โ€” Massive Text Embedding Benchmark. -- ๐Ÿงช ๐Ÿ†“ [**BEIR**](https://github.com/beir-cellar/beir) โ€” Zero-shot IR benchmark. - ๐Ÿงช ๐Ÿ†“ [**ARES**](https://github.com/stanford-futuredata/ARES) โ€” Automated RAG evaluation. +- ๐Ÿงช ๐Ÿ†“ [**BEIR**](https://github.com/beir-cellar/beir) โ€” Zero-shot IR benchmark. +- ๐Ÿงช ๐Ÿ†“ [**MTEB**](https://huggingface.co/spaces/mteb/leaderboard) โ€” Massive Text Embedding Benchmark. +- ๐Ÿงช ๐Ÿ†“ [**RAGAS**](https://docs.ragas.io/) โ€” Framework and leaderboard for RAG eval. ### T10 ยท Tool Use & Function Calling -- ๐Ÿงช ๐Ÿ†“ [**Berkeley Function-Calling Leaderboard (BFCL)**](https://gorilla.cs.berkeley.edu/leaderboard.html) โ€” UC Berkeley. -- ๐Ÿงช ๐Ÿ†“ [**ฯ„-bench**](https://github.com/sierra-research/tau-bench) โ€” Sierra. Tool-agent-user interaction benchmark. - ๐Ÿงช ๐Ÿ†“ [**API-Bank**](https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank) โ€” Alibaba. Tool-augmented assistants. +- ๐Ÿงช ๐Ÿ†“ [**ฯ„-bench**](https://github.com/sierra-research/tau-bench) โ€” Sierra. Tool-agent-user interaction benchmark. +- ๐Ÿงช ๐Ÿ†“ [**Berkeley Function-Calling Leaderboard (BFCL)**](https://gorilla.cs.berkeley.edu/leaderboard.html) โ€” UC Berkeley. ### T11 ยท Orchestration / T12 ยท Multi-Agent @@ -914,8 +914,8 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T13 ยท Evaluation -- ๐Ÿงช ๐Ÿ†“ [**HELM**](https://crfm.stanford.edu/helm/) โ€” Stanford CRFM. Holistic evaluation. - ๐Ÿงช ๐Ÿ†“ [**Chatbot Arena / LMSYS Arena**](https://lmarena.ai/) โ€” Human-preference leaderboard. +- ๐Ÿงช ๐Ÿ†“ [**HELM**](https://crfm.stanford.edu/helm/) โ€” Stanford CRFM. Holistic evaluation. - ๐Ÿงช ๐Ÿ†“ [**MMLU-Pro**](https://github.com/TIGER-AI-Lab/MMLU-Pro) โ€” Harder MMLU. - ๐Ÿงช ๐Ÿ†“ [**MT-Bench**](https://huggingface.co/spaces/lmsys/mt-bench) โ€” LLM-as-judge multi-turn. @@ -927,20 +927,20 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T16 ยท Safety & Alignment -- ๐Ÿงช ๐Ÿ†“ [**TruthfulQA**](https://github.com/sylinrl/TruthfulQA) โ€” Truthfulness benchmark. - ๐Ÿงช ๐Ÿ†“ [**BBQ**](https://github.com/nyu-mll/BBQ) โ€” Bias benchmark. - ๐Ÿงช ๐Ÿ†“ [**ToxiGen**](https://github.com/microsoft/TOXIGEN) โ€” Toxicity. +- ๐Ÿงช ๐Ÿ†“ [**TruthfulQA**](https://github.com/sylinrl/TruthfulQA) โ€” Truthfulness benchmark. ### T18 ยท Inference -- ๐Ÿงช ๐Ÿ†“ [**MLPerf Inference**](https://mlcommons.org/benchmarks/inference/) โ€” MLCommons. Industry-standard serving benchmark. - ๐Ÿงช ๐Ÿ†“ [**LLMPerf**](https://github.com/ray-project/llmperf) โ€” Anyscale. Throughput/latency tool. +- ๐Ÿงช ๐Ÿ†“ [**MLPerf Inference**](https://mlcommons.org/benchmarks/inference/) โ€” MLCommons. Industry-standard serving benchmark. ### T19 ยท Voice & Multimodal +- ๐Ÿงช ๐Ÿ†“ [**Dynabench speech**](https://dynabench.org/) โ€” Live speech-model benchmarks. - ๐Ÿงช ๐Ÿ†“ [**MMMU**](https://mmmu-benchmark.github.io/) โ€” Multimodal multidiscipline benchmark. - ๐Ÿงช ๐Ÿ†“ [**VideoMME**](https://video-mme.github.io/) โ€” Video understanding. -- ๐Ÿงช ๐Ÿ†“ [**Dynabench speech**](https://dynabench.org/) โ€” Live speech-model benchmarks. @@ -950,8 +950,8 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T1 / T3 ยท Coding Agents & IDE Rules -- โญ ๐Ÿ—๏ธ ๐Ÿ†“ [**Claude Code**](https://github.com/anthropics/claude-code) โ€” Anthropic's reference agentic CLI. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Aider**](https://github.com/Aider-AI/aider) โ€” Reference terminal coding agent with detailed engineering blog. +- โญ ๐Ÿ—๏ธ ๐Ÿ†“ [**Claude Code**](https://github.com/anthropics/claude-code) โ€” Anthropic's reference agentic CLI. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Cline**](https://github.com/cline/cline) โ€” Open-source autonomous coding agent. - ๐Ÿ—๏ธ ๐Ÿ†“ [**OpenHands**](https://github.com/All-Hands-AI/OpenHands) โ€” All Hands AI. Open-source autonomous SWE agent. @@ -961,22 +961,22 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T5 ยท Autonomous SWE Agents -- ๐Ÿ—๏ธ ๐Ÿ†“ [**SWE-agent**](https://github.com/princeton-nlp/SWE-agent) โ€” Princeton NLP. Reference agent for SWE-bench. -- ๐Ÿ—๏ธ ๐Ÿ†“ [**AutoCodeRover**](https://github.com/nus-apr/auto-code-rover) โ€” NUS. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Agentless**](https://github.com/OpenAutoCoder/Agentless) โ€” Minimal agentless baseline that beat prior agents on SWE-bench Lite. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**AutoCodeRover**](https://github.com/nus-apr/auto-code-rover) โ€” NUS. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**SWE-agent**](https://github.com/princeton-nlp/SWE-agent) โ€” Princeton NLP. Reference agent for SWE-bench. ### T6 ยท App Architecture +- ๐Ÿ—๏ธ ๐Ÿ†“ [**LangChain templates**](https://github.com/langchain-ai/langchain/tree/master/templates) โ€” Reference app scaffolds. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Open Interpreter**](https://github.com/OpenInterpreter/open-interpreter) โ€” Reference local code-execution agent. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Quivr**](https://github.com/QuivrHQ/quivr) โ€” Reference full-stack RAG assistant. -- ๐Ÿ—๏ธ ๐Ÿ†“ [**LangChain templates**](https://github.com/langchain-ai/langchain/tree/master/templates) โ€” Reference app scaffolds. ### T8 ยท RAG +- ๐Ÿ—๏ธ ๐Ÿ†“ [**GraphRAG**](https://github.com/microsoft/graphrag) โ€” Microsoft Research. - โญ ๐Ÿ—๏ธ ๐Ÿ†“ [**LlamaIndex**](https://github.com/run-llama/llama_index) โ€” Reference RAG framework; docs double as case studies. - ๐Ÿ—๏ธ ๐Ÿ†“ [**RAGFlow**](https://github.com/infiniflow/ragflow) โ€” Production-grade RAG reference. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Verba**](https://github.com/weaviate/Verba) โ€” Weaviate reference RAG app. -- ๐Ÿ—๏ธ ๐Ÿ†“ [**GraphRAG**](https://github.com/microsoft/graphrag) โ€” Microsoft Research. ### T9 ยท Memory @@ -986,26 +986,26 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T10 ยท Tool Use & MCP -- ๐Ÿ—๏ธ ๐Ÿ†“ [**awesome-mcp-servers**](https://github.com/punkpeye/awesome-mcp-servers) โ€” Community catalogue of MCP server implementations. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Anthropic MCP reference servers**](https://github.com/modelcontextprotocol/servers) โ€” The canonical reference MCP servers. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**awesome-mcp-servers**](https://github.com/punkpeye/awesome-mcp-servers) โ€” Community catalogue of MCP server implementations. ### T11 / T12 ยท Orchestration & Multi-Agent -- ๐Ÿ—๏ธ ๐Ÿ†“ [**LangGraph**](https://github.com/langchain-ai/langgraph) โ€” Reference graph-based orchestration. - ๐Ÿ—๏ธ ๐Ÿ†“ [**AutoGen**](https://github.com/microsoft/autogen) โ€” Microsoft. - ๐Ÿ—๏ธ ๐Ÿ†“ [**CrewAI**](https://github.com/crewAIInc/crewAI) โ€” Reference role-based multi-agent. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**LangGraph**](https://github.com/langchain-ai/langgraph) โ€” Reference graph-based orchestration. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Pydantic AI**](https://github.com/pydantic/pydantic-ai) โ€” Type-safe agent framework. ### T13 ยท Evaluation -- ๐Ÿ—๏ธ ๐Ÿ†“ [**EleutherAI lm-evaluation-harness**](https://github.com/EleutherAI/lm-evaluation-harness) โ€” Standard offline-eval harness. - ๐Ÿ—๏ธ ๐Ÿ†“ [**DeepEval**](https://github.com/confident-ai/deepeval) โ€” Reference eval framework. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**EleutherAI lm-evaluation-harness**](https://github.com/EleutherAI/lm-evaluation-harness) โ€” Standard offline-eval harness. - ๐Ÿ—๏ธ ๐Ÿ†“ [**RAGAS**](https://github.com/explodinggradients/ragas) โ€” RAG-specific evaluation. ### T14 ยท Observability -- ๐Ÿ—๏ธ ๐Ÿ†“ [**Langfuse**](https://github.com/langfuse/langfuse) โ€” Open-source LLM observability. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Arize Phoenix**](https://github.com/Arize-ai/phoenix) โ€” Open-source tracing + evals. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**Langfuse**](https://github.com/langfuse/langfuse) โ€” Open-source LLM observability. - ๐Ÿ—๏ธ ๐Ÿ†“ [**OpenLLMetry**](https://github.com/traceloop/openllmetry) โ€” OTel-based LLM instrumentation. ### T15 ยท Guardrails & Security @@ -1016,17 +1016,17 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T17 ยท Fine-tuning -- ๐Ÿ—๏ธ ๐Ÿ†“ [**Unsloth**](https://github.com/unslothai/unsloth) โ€” Fast LoRA/QLoRA reference. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Axolotl**](https://github.com/axolotl-ai-cloud/axolotl) โ€” Reference fine-tuning framework. -- ๐Ÿ—๏ธ ๐Ÿ†“ [**LLaMA-Factory**](https://github.com/hiyouga/LLaMA-Factory) โ€” Unified fine-tuning toolkit. - ๐Ÿ—๏ธ ๐Ÿ†“ [**Hugging Face alignment-handbook**](https://github.com/huggingface/alignment-handbook) โ€” Reference RLHF/DPO recipes. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**LLaMA-Factory**](https://github.com/hiyouga/LLaMA-Factory) โ€” Unified fine-tuning toolkit. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**Unsloth**](https://github.com/unslothai/unsloth) โ€” Fast LoRA/QLoRA reference. ### T18 ยท Inference & Serving -- โญ ๐Ÿ—๏ธ ๐Ÿ†“ [**vLLM**](https://github.com/vllm-project/vllm) โ€” Reference high-throughput LLM serving. -- ๐Ÿ—๏ธ ๐Ÿ†“ [**SGLang**](https://github.com/sgl-project/sglang) โ€” Structured generation serving. - ๐Ÿ—๏ธ ๐Ÿ†“ [**llama.cpp**](https://github.com/ggerganov/llama.cpp) โ€” Reference CPU/GPU local inference. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**SGLang**](https://github.com/sgl-project/sglang) โ€” Structured generation serving. - ๐Ÿ—๏ธ ๐Ÿ†“ [**TensorRT-LLM**](https://github.com/NVIDIA/TensorRT-LLM) โ€” NVIDIA reference optimised serving. +- โญ ๐Ÿ—๏ธ ๐Ÿ†“ [**vLLM**](https://github.com/vllm-project/vllm) โ€” Reference high-throughput LLM serving. ### T19 ยท Voice & Multimodal @@ -1036,9 +1036,9 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T20 ยท Product & UX -- ๐Ÿ—๏ธ ๐Ÿ†“ [**Vercel AI SDK**](https://github.com/vercel/ai) โ€” Reference AI-UI patterns and streaming. -- ๐Ÿ—๏ธ ๐Ÿ†“ [**Open WebUI**](https://github.com/open-webui/open-webui) โ€” Reference local chat UI. - ๐Ÿ—๏ธ ๐Ÿ†“ [**assistant-ui**](https://github.com/assistant-ui/assistant-ui) โ€” Reference React components for AI chat. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**Open WebUI**](https://github.com/open-webui/open-webui) โ€” Reference local chat UI. +- ๐Ÿ—๏ธ ๐Ÿ†“ [**Vercel AI SDK**](https://github.com/vercel/ai) โ€” Reference AI-UI patterns and streaming. @@ -1049,23 +1049,23 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### Conference series - โญ ๐ŸŽฅ ๐Ÿ†“ [**AI Engineer Summit / World's Fair**](https://www.ai.engineer/) โ€” The definitive practitioner conference; full talks on YouTube. -- ๐ŸŽฅ ๐Ÿ†“ [**NeurIPS / ICML / ICLR**](https://neurips.cc/) โ€” Core ML research venues; most papers include recorded talks. - ๐ŸŽฅ ๐Ÿ†“ [**COLM**](https://colmweb.org/) โ€” Conference on Language Modeling. New dedicated LM venue. -- ๐ŸŽฅ ๐Ÿ†“ [**MLSys**](https://mlsys.org/) โ€” Core ML-systems conference (inference, serving). - ๐ŸŽฅ ๐Ÿ†“ [**LlamaCon**](https://www.llama.com/events/llamacon/) โ€” Meta's open-source LLM conference. +- ๐ŸŽฅ ๐Ÿ†“ [**MLSys**](https://mlsys.org/) โ€” Core ML-systems conference (inference, serving). +- ๐ŸŽฅ ๐Ÿ†“ [**NeurIPS / ICML / ICLR**](https://neurips.cc/) โ€” Core ML research venues; most papers include recorded talks. ### Canonical talks +- โญ ๐ŸŽฅ ๐Ÿ†“ [**1hr Talk: Intro to LLMs (Nov 2024)**](https://www.youtube.com/watch?v=7xTGNNLPyMI) โ€” Karpathy updated "Deep Dive into LLMs". - โญ ๐ŸŽฅ ๐Ÿ†“ [**Intro to LLMs**](https://www.youtube.com/watch?v=zjkBMFhNj_g) โ€” Andrej Karpathy. The reference "how LLMs work" talk. - โญ ๐ŸŽฅ ๐Ÿ†“ [**Let's build GPT: from scratch, in code**](https://www.youtube.com/watch?v=kCc8FmEb1nY) โ€” Andrej Karpathy. -- โญ ๐ŸŽฅ ๐Ÿ†“ [**1hr Talk: Intro to LLMs (Nov 2024)**](https://www.youtube.com/watch?v=7xTGNNLPyMI) โ€” Karpathy updated "Deep Dive into LLMs". -- ๐ŸŽฅ ๐Ÿ†“ [**State of GPT**](https://www.youtube.com/watch?v=bZQun8Y4L2A) โ€” Andrej Karpathy (Microsoft Build 2023). - ๐ŸŽฅ ๐Ÿ†“ [**Stanford CS25: Transformers United**](https://web.stanford.edu/class/cs25/) โ€” Full lecture series. +- ๐ŸŽฅ ๐Ÿ†“ [**State of GPT**](https://www.youtube.com/watch?v=bZQun8Y4L2A) โ€” Andrej Karpathy (Microsoft Build 2023). ### T1 ยท Coding Agents -- ๐ŸŽฅ ๐Ÿ†“ [**Mastering Claude Code**](https://www.youtube.com/watch?v=r-ML9gZZBVo) โ€” Anthropic (Boris Cherny). - ๐ŸŽฅ ๐Ÿ†“ [**Cursor: Building the AI-first IDE**](https://www.youtube.com/@cursor-ai) โ€” Cursor team channel. +- ๐ŸŽฅ ๐Ÿ†“ [**Mastering Claude Code**](https://www.youtube.com/watch?v=r-ML9gZZBVo) โ€” Anthropic (Boris Cherny). - ๐ŸŽฅ ๐Ÿ†“ [**The future of AI coding**](https://www.youtube.com/@latent-space) โ€” Latent Space talk archives. ### T4 ยท SWE Benchmarks @@ -1074,8 +1074,8 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T6 ยท App Architecture -- ๐ŸŽฅ ๐Ÿ†“ [**State of AI Engineering**](https://www.youtube.com/@latent-space) โ€” Latent Space keynotes. - ๐ŸŽฅ ๐Ÿ†“ [**Emerging architectures for LLM applications**](https://a16z.com/emerging-architectures-for-llm-applications/) โ€” a16z (video + post). +- ๐ŸŽฅ ๐Ÿ†“ [**State of AI Engineering**](https://www.youtube.com/@latent-space) โ€” Latent Space keynotes. ### T7 ยท Prompt Engineering @@ -1084,13 +1084,13 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T8 ยท RAG -- ๐ŸŽฅ ๐Ÿ†“ [**Systematically improving RAG applications**](https://www.youtube.com/watch?v=RrDBV9W4zTc) โ€” Jason Liu. - ๐ŸŽฅ ๐Ÿ†“ [**RAG at scale**](https://www.youtube.com/@LangChain) โ€” LangChain channel series. +- ๐ŸŽฅ ๐Ÿ†“ [**Systematically improving RAG applications**](https://www.youtube.com/watch?v=RrDBV9W4zTc) โ€” Jason Liu. ### T10 ยท MCP -- ๐ŸŽฅ ๐Ÿ†“ [**Model Context Protocol deep dive**](https://www.youtube.com/@AnthropicAI) โ€” Anthropic. - ๐ŸŽฅ ๐Ÿ†“ [**MCP at AI Engineer Summit**](https://www.youtube.com/@aiDotEngineer) โ€” AI Engineer. +- ๐ŸŽฅ ๐Ÿ†“ [**Model Context Protocol deep dive**](https://www.youtube.com/@AnthropicAI) โ€” Anthropic. ### T11 / T12 ยท Orchestration & Multi-Agent @@ -1108,23 +1108,23 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### T15 / T16 ยท Security & Safety -- ๐ŸŽฅ ๐Ÿ†“ [**Simon Willison on prompt injection**](https://simonwillison.net/tags/prompt-injection/) โ€” Talks + essays hub. - ๐ŸŽฅ ๐Ÿ†“ [**Anthropic AI safety research**](https://www.youtube.com/@AnthropicAI) โ€” Anthropic channel. +- ๐ŸŽฅ ๐Ÿ†“ [**Simon Willison on prompt injection**](https://simonwillison.net/tags/prompt-injection/) โ€” Talks + essays hub. ### T17 ยท Fine-tuning -- ๐ŸŽฅ ๐Ÿ†“ [**Let's reproduce GPT-2 / build the GPT tokenizer**](https://www.youtube.com/@AndrejKarpathy) โ€” Karpathy channel. - ๐ŸŽฅ ๐Ÿ†“ [**Fine-tuning workshop**](https://www.youtube.com/@parlance-labs) โ€” Hamel Husain channel. +- ๐ŸŽฅ ๐Ÿ†“ [**Let's reproduce GPT-2 / build the GPT tokenizer**](https://www.youtube.com/@AndrejKarpathy) โ€” Karpathy channel. ### T18 ยท Inference -- ๐ŸŽฅ ๐Ÿ†“ [**vLLM: high-throughput LLM serving**](https://www.youtube.com/@anyscale) โ€” Anyscale / UC Berkeley talks. - ๐ŸŽฅ ๐Ÿ†“ [**CUDA Mode lectures**](https://github.com/cuda-mode/lectures) โ€” Community GPU/kernel series. +- ๐ŸŽฅ ๐Ÿ†“ [**vLLM: high-throughput LLM serving**](https://www.youtube.com/@anyscale) โ€” Anyscale / UC Berkeley talks. ### T19 ยท Voice & Multimodal -- ๐ŸŽฅ ๐Ÿ†“ [**OpenAI Realtime API demos**](https://www.youtube.com/@OpenAI) โ€” OpenAI. - ๐ŸŽฅ ๐Ÿ†“ [**LiveKit voice-agent talks**](https://www.youtube.com/@livekitio) โ€” LiveKit. +- ๐ŸŽฅ ๐Ÿ†“ [**OpenAI Realtime API demos**](https://www.youtube.com/@OpenAI) โ€” OpenAI. ### T20 ยท Product & UX @@ -1142,19 +1142,19 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR *Recurring podcasts with strong agentic & AI-engineering coverage.* -- โญ ๐ŸŽง ๐Ÿ†“ [**Latent Space**](https://www.latent.space/podcast) โ€” swyx & Alessio. The AI-engineering podcast of record; guests include most major AI-lab engineers. -- โญ ๐ŸŽง ๐Ÿ†“ [**Practical AI**](https://changelog.com/practicalai) โ€” Daniel Whitenack & Chris Benson. Long-running, practitioner-first. -- ๐ŸŽง ๐Ÿ†“ [**MLOps Community podcast**](https://mlops.community/podcast/) โ€” Demetrios Brinkmann. Ops-side operationalisation case studies. -- ๐ŸŽง ๐Ÿ†“ [**Gradient Dissent**](https://wandb.ai/fully-connected/podcast) โ€” Weights & Biases. Applied-ML interviews. -- ๐ŸŽง ๐Ÿ†“ [**The TWIML AI Podcast**](https://twimlai.com/podcast/twimlai/) โ€” Sam Charrington. Longest-running ML interview series. -- ๐ŸŽง ๐Ÿ†“ [**No Priors**](https://www.no-priors.com/) โ€” Sarah Guo & Elad Gil. Founders / researchers. - ๐ŸŽง ๐Ÿ†“ [**Cognitive Revolution**](https://www.cognitiverevolution.ai/) โ€” Nathan Labenz. Weekly AI engineering + strategy. - ๐ŸŽง ๐Ÿ†“ [**Dwarkesh Podcast**](https://www.dwarkeshpatel.com/podcast) โ€” Dwarkesh Patel. Deep interviews with top researchers. -- ๐ŸŽง ๐Ÿ†“ [**Machine Learning Street Talk**](https://www.youtube.com/@MachineLearningStreetTalk) โ€” Tim Scarfe. Technical deep-dives. -- ๐ŸŽง ๐Ÿ†“ [**Lex Fridman Podcast**](https://lexfridman.com/podcast/) โ€” Long-form interviews with AI-lab CEOs and researchers. -- ๐ŸŽง ๐Ÿ†“ [**Unsupervised Learning**](https://unsupervisedlearning.com/) โ€” Redpoint. AI-founder / operator conversations. +- ๐ŸŽง ๐Ÿ†“ [**Gradient Dissent**](https://wandb.ai/fully-connected/podcast) โ€” Weights & Biases. Applied-ML interviews. - ๐ŸŽง ๐Ÿ†“ [**Interconnects**](https://www.interconnects.ai/podcast) โ€” Nathan Lambert. RLHF / post-training focus. +- โญ ๐ŸŽง ๐Ÿ†“ [**Latent Space**](https://www.latent.space/podcast) โ€” swyx & Alessio. The AI-engineering podcast of record; guests include most major AI-lab engineers. +- ๐ŸŽง ๐Ÿ†“ [**Lex Fridman Podcast**](https://lexfridman.com/podcast/) โ€” Long-form interviews with AI-lab CEOs and researchers. +- ๐ŸŽง ๐Ÿ†“ [**Machine Learning Street Talk**](https://www.youtube.com/@MachineLearningStreetTalk) โ€” Tim Scarfe. Technical deep-dives. +- ๐ŸŽง ๐Ÿ†“ [**MLOps Community podcast**](https://mlops.community/podcast/) โ€” Demetrios Brinkmann. Ops-side operationalisation case studies. +- ๐ŸŽง ๐Ÿ†“ [**No Priors**](https://www.no-priors.com/) โ€” Sarah Guo & Elad Gil. Founders / researchers. +- โญ ๐ŸŽง ๐Ÿ†“ [**Practical AI**](https://changelog.com/practicalai) โ€” Daniel Whitenack & Chris Benson. Long-running, practitioner-first. - ๐ŸŽง ๐Ÿ†“ [**Pragmatic Engineer**](https://newsletter.pragmaticengineer.com/podcast) โ€” Gergely Orosz. AI-engineering org/hiring coverage. +- ๐ŸŽง ๐Ÿ†“ [**The TWIML AI Podcast**](https://twimlai.com/podcast/twimlai/) โ€” Sam Charrington. Longest-running ML interview series. +- ๐ŸŽง ๐Ÿ†“ [**Unsupervised Learning**](https://unsupervisedlearning.com/) โ€” Redpoint. AI-founder / operator conversations. @@ -1162,23 +1162,23 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR *Weekly and monthly curated newsletters.* -- โญ ๐Ÿ“ฐ ๐Ÿ†“ [**The Batch**](https://www.deeplearning.ai/the-batch/) โ€” Andrew Ng / DeepLearning.AI. Weekly AI-engineering digest. -- โญ ๐Ÿ“ฐ ๐Ÿ†“ [**Import AI**](https://importai.substack.com/) โ€” Jack Clark (Anthropic co-founder). Policy + research. -- โญ ๐Ÿ“ฐ ๐Ÿ†“ [**Latent Space**](https://www.latent.space/) โ€” swyx. The AI-engineering newsletter of record. -- ๐Ÿ“ฐ ๐Ÿ†“ [**Simon Willison's Weblog**](https://simonwillison.net/) โ€” RSS/email. Daily real-time coverage of tools and agents. - ๐Ÿ“ฐ ๐Ÿ†“ [**Ahead of AI**](https://magazine.sebastianraschka.com/) โ€” Sebastian Raschka. LLM research + fine-tuning deep-dives. -- ๐Ÿ“ฐ ๐Ÿ†“ [**The Pragmatic Engineer**](https://newsletter.pragmaticengineer.com/) โ€” Gergely Orosz. AI-engineering hiring/org coverage. -- ๐Ÿ“ฐ ๐Ÿ†“ [**Interconnects**](https://www.interconnects.ai/) โ€” Nathan Lambert. RLHF / post-training. -- ๐Ÿ“ฐ ๐Ÿ†“ [**Last Week in AI**](https://lastweekin.ai/) โ€” Weekly recap. -- ๐Ÿ“ฐ ๐Ÿ†“ [**TLDR AI**](https://tldr.tech/ai) โ€” Daily headlines. - ๐Ÿ“ฐ ๐Ÿ†“ [**Ben's Bites**](https://bensbites.co/) โ€” Daily digest; founder-friendly. - ๐Ÿ“ฐ ๐Ÿ†“ [**Chip Huyen's Blog**](https://huyenchip.com/blog/) โ€” Occasional long-form on AI engineering. +- ๐Ÿ“ฐ ๐Ÿ†“ [**DiamantAI**](https://diamantai.substack.com) โ€” Nir Diamant. Practical AI engineering and generative AI: RAG, agents, and LLM application patterns explained simply. - ๐Ÿ“ฐ ๐Ÿ†“ [**Eugene Yan**](https://eugeneyan.com/) โ€” Pattern / eval / RAG deep-dives. - ๐Ÿ“ฐ ๐Ÿ†“ [**Hamel's Blog**](https://hamel.dev/) โ€” Evals + applied LLMs. +- โญ ๐Ÿ“ฐ ๐Ÿ†“ [**Import AI**](https://importai.substack.com/) โ€” Jack Clark (Anthropic co-founder). Policy + research. +- ๐Ÿ“ฐ ๐Ÿ†“ [**Interconnects**](https://www.interconnects.ai/) โ€” Nathan Lambert. RLHF / post-training. +- ๐Ÿ“ฐ ๐Ÿ†“ [**Last Week in AI**](https://lastweekin.ai/) โ€” Weekly recap. +- โญ ๐Ÿ“ฐ ๐Ÿ†“ [**Latent Space**](https://www.latent.space/) โ€” swyx. The AI-engineering newsletter of record. - ๐Ÿ“ฐ ๐Ÿ†“ [**Machine Learning Engineer Newsletter**](https://ethical.institute/mle.html) โ€” Alejandro Saucedo. Weekly production-ML curation. - ๐Ÿ“ฐ ๐Ÿ†“ [**MLOps Community newsletter**](https://mlops.community/) โ€” MLOps Community. +- ๐Ÿ“ฐ ๐Ÿ†“ [**Simon Willison's Weblog**](https://simonwillison.net/) โ€” RSS/email. Daily real-time coverage of tools and agents. +- โญ ๐Ÿ“ฐ ๐Ÿ†“ [**The Batch**](https://www.deeplearning.ai/the-batch/) โ€” Andrew Ng / DeepLearning.AI. Weekly AI-engineering digest. - ๐Ÿ“ฐ ๐Ÿ†“ [**The Data Exchange**](https://thedataexchange.media/) โ€” Ben Lorica. -- ๐Ÿ“ฐ ๐Ÿ†“ [**DiamantAI**](https://diamantai.substack.com) - Nir Diamant. Practical AI engineering and generative AI: RAG, agents, and LLM application patterns explained simply. +- ๐Ÿ“ฐ ๐Ÿ†“ [**The Pragmatic Engineer**](https://newsletter.pragmaticengineer.com/) โ€” Gergely Orosz. AI-engineering hiring/org coverage. +- ๐Ÿ“ฐ ๐Ÿ†“ [**TLDR AI**](https://tldr.tech/ai) โ€” Daily headlines. @@ -1188,32 +1188,32 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### Policy & frameworks +- ๐Ÿ†“ [**EU AI Act**](https://artificialintelligenceact.eu/) โ€” European Commission. Official text + implementation timeline. - โญ ๐Ÿ†“ [**NIST AI Risk Management Framework (AI RMF 1.0)**](https://www.nist.gov/itl/ai-risk-management-framework) โ€” NIST. The foundational US framework. - ๐Ÿ†“ [**NIST Generative AI Profile (NIST-AI-600-1)**](https://www.nist.gov/itl/ai-risk-management-framework) โ€” NIST. -- ๐Ÿ†“ [**EU AI Act**](https://artificialintelligenceact.eu/) โ€” European Commission. Official text + implementation timeline. -- ๐Ÿ†“ [**UK AI Safety Institute reports**](https://www.gov.uk/government/organisations/ai-safety-institute) โ€” UK AISI. - ๐Ÿ†“ [**OECD AI Principles**](https://oecd.ai/en/ai-principles) โ€” International reference. +- ๐Ÿ†“ [**UK AI Safety Institute reports**](https://www.gov.uk/government/organisations/ai-safety-institute) โ€” UK AISI. ### Lab safety & responsible scaling -- โญ ๐Ÿ†“ [**Anthropic Responsible Scaling Policy**](https://www.anthropic.com/news/anthropics-responsible-scaling-policy) โ€” Anthropic. - ๐Ÿ†“ [**Anthropic Core Views on AI Safety**](https://www.anthropic.com/news/core-views-on-ai-safety) โ€” Anthropic. -- ๐Ÿ†“ [**OpenAI Preparedness Framework**](https://openai.com/safety/preparedness) โ€” OpenAI. +- โญ ๐Ÿ†“ [**Anthropic Responsible Scaling Policy**](https://www.anthropic.com/news/anthropics-responsible-scaling-policy) โ€” Anthropic. - ๐Ÿ†“ [**Google DeepMind: Frontier Safety Framework**](https://deepmind.google/discover/blog/introducing-the-frontier-safety-framework/) โ€” Google DeepMind. +- ๐Ÿ†“ [**OpenAI Preparedness Framework**](https://openai.com/safety/preparedness) โ€” OpenAI. ### Security & red-teaming -- โญ ๐Ÿ†“ [**OWASP Top 10 for LLM Applications**](https://genai.owasp.org/llm-top-10/) โ€” OWASP. +- ๐Ÿ†“ [**HarmBench**](https://www.harmbench.org/) โ€” CAIS. - ๐Ÿ†“ [**MITRE ATLAS**](https://atlas.mitre.org/) โ€” Adversarial threat landscape for AI systems. - ๐Ÿ†“ [**NIST Adversarial ML Taxonomy (NIST AI 100-2)**](https://csrc.nist.gov/pubs/ai/100/2/e2023/final) โ€” NIST. -- ๐Ÿ†“ [**HarmBench**](https://www.harmbench.org/) โ€” CAIS. +- โญ ๐Ÿ†“ [**OWASP Top 10 for LLM Applications**](https://genai.owasp.org/llm-top-10/) โ€” OWASP. - ๐Ÿ†“ [**Simon Willison's prompt-injection series**](https://simonwillison.net/series/prompt-injection/) โ€” SW. ### Responsible AI practice -- ๐Ÿ†“ [**Microsoft Responsible AI Standard**](https://www.microsoft.com/en-us/ai/responsible-ai) โ€” Microsoft. -- ๐Ÿ†“ [**Google Responsible AI practices**](https://ai.google/responsibility/responsible-ai-practices/) โ€” Google. - ๐Ÿ†“ [**Fairlearn**](https://fairlearn.org/) โ€” Open-source fairness toolkit. +- ๐Ÿ†“ [**Google Responsible AI practices**](https://ai.google/responsibility/responsible-ai-practices/) โ€” Google. +- ๐Ÿ†“ [**Microsoft Responsible AI Standard**](https://www.microsoft.com/en-us/ai/responsible-ai) โ€” Microsoft. - ๐Ÿ†“ [**Partnership on AI**](https://partnershiponai.org/) โ€” Multi-stakeholder org with published frameworks and incident database. ### Papers & research @@ -1231,20 +1231,20 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR ### Design & UX -- โญ ๐Ÿ†“ [**Guidelines for Human-AI Interaction**](https://www.microsoft.com/en-us/research/publication/guidelines-for-human-ai-interaction/) โ€” Amershi et al. (Microsoft Research). The canonical design heuristics. -- ๐Ÿ†“ [**NNGroup: Generative AI design patterns**](https://www.nngroup.com/articles/generative-ai-design-patterns/) โ€” Nielsen Norman Group. -- ๐Ÿ†“ [**Google's People + AI Guidebook**](https://pair.withgoogle.com/guidebook/) โ€” Google PAIR. - ๐Ÿ†“ [**Apple Human Interface Guidelines โ€” Generative AI**](https://developer.apple.com/design/human-interface-guidelines/machine-learning) โ€” Apple. -- ๐Ÿ†“ [**Maggie Appleton**](https://maggieappleton.com/) โ€” Essays on the UX of agentic, malleable software. +- ๐Ÿ†“ [**Google's People + AI Guidebook**](https://pair.withgoogle.com/guidebook/) โ€” Google PAIR. +- โญ ๐Ÿ†“ [**Guidelines for Human-AI Interaction**](https://www.microsoft.com/en-us/research/publication/guidelines-for-human-ai-interaction/) โ€” Amershi et al. (Microsoft Research). The canonical design heuristics. - ๐Ÿ†“ [**Linus Lee**](https://thesephist.com/) โ€” Essays on interfaces for tools of thought. +- ๐Ÿ†“ [**Maggie Appleton**](https://maggieappleton.com/) โ€” Essays on the UX of agentic, malleable software. +- ๐Ÿ†“ [**NNGroup: Generative AI design patterns**](https://www.nngroup.com/articles/generative-ai-design-patterns/) โ€” Nielsen Norman Group. ### Economics & business - โญ ๐Ÿ†“ [**a16z: The Economic Case for Generative AI**](https://a16z.com/the-economic-case-for-generative-ai-and-foundation-models/) โ€” a16z. +- ๐Ÿ†“ [**Artificial Analysis**](https://artificialanalysis.ai/) โ€” Cross-provider pricing/latency/quality dashboards. +- ๐Ÿ†“ [**Epoch AI**](https://epochai.org/) โ€” Data on compute, cost, and scaling trends. - ๐Ÿ†“ [**Latent Space on unit economics**](https://www.latent.space/p/state-of-ai-engineering) โ€” Latent Space. - ๐Ÿ†“ [**Stanford AI Index Report**](https://aiindex.stanford.edu/report/) โ€” Stanford HAI. Annual deep economic + research snapshot. -- ๐Ÿ†“ [**Epoch AI**](https://epochai.org/) โ€” Data on compute, cost, and scaling trends. -- ๐Ÿ†“ [**Artificial Analysis**](https://artificialanalysis.ai/) โ€” Cross-provider pricing/latency/quality dashboards. ### Product strategy @@ -1258,17 +1258,17 @@ Please review our [CONTRIBUTING.md](./CONTRIBUTING.md) before submitting a PR *How organisations structure AI-engineering work, hire for it, and operate sustainably.* -- โญ ๐Ÿ†“ [**The Pragmatic Engineer โ€” AI tag**](https://newsletter.pragmaticengineer.com/t/ai) โ€” Gergely Orosz. AI-engineering hiring + org design. +- ๐Ÿ†“ [**a16z AI canon**](https://a16z.com/ai-canon/) โ€” a16z. Curated reading list for people building AI teams. - ๐Ÿ†“ [**Building the AI Engineer role**](https://www.latent.space/p/ai-engineer) โ€” swyx / Latent Space. The foundational essay defining "AI Engineer" as a discipline. -- ๐Ÿ†“ [**What is an AI Engineer?**](https://applied-llms.org/) โ€” Applied LLMs consortium. +- ๐Ÿ†“ [**Chip Huyen: Machine learning in production**](https://huyenchip.com/2022/01/02/real-world-mlops-questions.html) โ€” Org-design questions from production ML. +- ๐Ÿ†“ [**DeepLearning.AI AI Engineer Hiring Report**](https://www.deeplearning.ai/the-batch/) โ€” The Batch periodic coverage. +- ๐Ÿ†“ [**Emmanuel Ameisen: Building ML Powered Applications**](https://mlpowered.com/book/) โ€” Book + blog on AI-team building. - ๐Ÿ†“ [**Eugene Yan: Team size and velocity**](https://eugeneyan.com/writing/team-size/) โ€” Eugene Yan. +- ๐Ÿ†“ [**GitHub: The AI-native developer**](https://github.blog/ai-and-ml/) โ€” GitHub's research on workflows / productivity. - ๐Ÿ†“ [**Shreya Shankar: Operationalizing ML**](https://www.shreya-shankar.com/) โ€” Shreya Shankar. - ๐Ÿ†“ [**Staff Engineer โ€” AI org posts**](https://staffeng.com/) โ€” Will Larson & community. -- ๐Ÿ†“ [**a16z AI canon**](https://a16z.com/ai-canon/) โ€” a16z. Curated reading list for people building AI teams. -- ๐Ÿ†“ [**Emmanuel Ameisen: Building ML Powered Applications**](https://mlpowered.com/book/) โ€” Book + blog on AI-team building. -- ๐Ÿ†“ [**DeepLearning.AI AI Engineer Hiring Report**](https://www.deeplearning.ai/the-batch/) โ€” The Batch periodic coverage. -- ๐Ÿ†“ [**Chip Huyen: Machine learning in production**](https://huyenchip.com/2022/01/02/real-world-mlops-questions.html) โ€” Org-design questions from production ML. -- ๐Ÿ†“ [**GitHub: The AI-native developer**](https://github.blog/ai-and-ml/) โ€” GitHub's research on workflows / productivity. +- โญ ๐Ÿ†“ [**The Pragmatic Engineer โ€” AI tag**](https://newsletter.pragmaticengineer.com/t/ai) โ€” Gergely Orosz. AI-engineering hiring + org design. +- ๐Ÿ†“ [**What is an AI Engineer?**](https://applied-llms.org/) โ€” Applied LLMs consortium. ---