Skip to content

Observability Hooks

Jump to bottom
skobeltsyn edited this page May 12, 2026 · 3 revisions

Observability Hooks

Monitor what your agents do at runtime -- tool calls, knowledge fetches, and skill selections -- without modifying business logic.

Overview

Agents.KT provides lightweight hooks on the Agent type. Each fires at a specific point in the agent's execution lifecycle:

Hook Fires When Signature
onToolUse After a tool executor completes (name: String, args: Map<String, Any?>, result: Any?) -> Unit
onKnowledgeUsed When the LLM fetches a knowledge entry (name: String, content: String) -> Unit
onSkillChosen After skill selection resolves (name: String) -> Unit
onError Infrastructure error is about to propagate (error: Throwable) -> Unit
onBudgetThreshold A cumulative budget crosses a warning threshold (reason: BudgetReason, usedPercent: Double) -> Unit
observe Unified typed stream for skill/tool/knowledge/error events (event: PipelineEvent) -> Unit

All hooks are optional. They do not affect execution -- the agent runs identically with or without them. They are purely observational; the original error still rethrows after onError runs.

onToolUse

Fires after every tool execution within the agentic loop. You receive the tool name, the arguments the LLM provided, and the result the executor returned.

val agent = agent<String, String>("file-ops") {
    model { ollama("qwen2.5:7b") }
    budget { maxTurns = 10 }

    lateinit var readFile: Tool<Map<String, Any?>, Any?>
    lateinit var writeFile: Tool<Map<String, Any?>, Any?>

    tools {
        readFile = tool("read_file", "Read a file") { args ->
            File(args["path"] as String).readText()
        }
        writeFile = tool("write_file", "Write a file") { args ->
            val path = args["path"] as String
            val content = args["content"] as String
            File(path).writeText(content)
            "Written ${content.length} bytes"
        }
    }

    skills {
        skill<String, String>("manage", "Manage files") {
            tools(readFile, writeFile)
        }
    }

    onToolUse { name, args, result ->
        println("TOOL [$name] args=$args result=$result")
    }
}

Output when the agent reads and writes a file:

TOOL [read_file] args={path=/tmp/input.txt} result=Hello, World!
TOOL [write_file] args={path=/tmp/output.txt, content=HELLO, WORLD!} result=Written 13 bytes

Use Cases

Structured logging:

onToolUse { name, args, result ->
    logger.info(
        "tool_call",
        mapOf(
            "tool" to name,
            "args" to args,
            "result_type" to result?.javaClass?.simpleName,
            "result_length" to result?.toString()?.length
        )
    )
}

Metrics collection:

onToolUse { name, args, result ->
    metrics.counter("agent.tool.calls", "tool" to name).increment()
    metrics.timer("agent.tool.duration", "tool" to name).record(duration)
}

Audit trail:

val auditLog = mutableListOf<AuditEntry>()

onToolUse { name, args, result ->
    auditLog.add(AuditEntry(
        timestamp = Instant.now(),
        tool = name,
        args = args,
        result = result?.toString()
    ))
}

onKnowledgeUsed

Fires when the LLM decides to fetch a knowledge entry during an agentic skill. You receive the knowledge entry's name and its content.

val agent = agent<String, String>("support-bot") {
    model { ollama("qwen2.5:7b") }
    budget { maxTurns = 5 }

    lateinit var searchDb: Tool<Map<String, Any?>, Any?>
    tools {
        searchDb = tool("search_db", "Search the support database") { args ->
            supportDb.search(args["query"] as String)
        }
    }

    skills {
        skill<String, String>("answer", "Answer questions") {
            tools(searchDb)

            knowledge("faq", "Frequently asked questions") {
                loadText("/data/faq.txt")
            }

            knowledge("pricing", "Current pricing information") {
                loadText("/data/pricing.txt")
            }
        }
    }

    onKnowledgeUsed { name, content ->
        println("KNOWLEDGE [$name] loaded ${content.length} chars")
    }
}

Output when the LLM fetches the FAQ:

KNOWLEDGE [faq] loaded 4823 chars

Use Cases

Track knowledge relevance:

val knowledgeHits = mutableMapOf<String, Int>()

onKnowledgeUsed { name, content ->
    knowledgeHits.merge(name, 1, Int::plus)
}

// After many runs:
// knowledgeHits = {faq=142, pricing=87, policies=23}
// "policies" is rarely used -- consider removing or restructuring it

Monitor knowledge freshness:

onKnowledgeUsed { name, content ->
    val lastUpdated = knowledgeMetadata[name]?.lastUpdated
    if (lastUpdated != null && lastUpdated.isBefore(Instant.now().minus(30, ChronoUnit.DAYS))) {
        logger.warn("Knowledge '$name' is over 30 days old -- consider refreshing")
    }
}

onSkillChosen

Fires after skill selection resolves, regardless of which strategy was used (predicate, LLM routing, or first-match). You receive the name of the chosen skill.

val agent = agent<String, String>("router") {
    model { ollama("qwen2.5:7b") }

    skills {
        skill<String, String>("billing", "Billing questions") { /* ... */ }
        skill<String, String>("technical", "Technical support") { /* ... */ }
        skill<String, String>("general", "General inquiries") { /* ... */ }
    }

    onSkillChosen { name ->
        println("ROUTING -> $name")
    }
}

Output:

ROUTING -> billing

Use Cases

Routing analytics:

onSkillChosen { name ->
    metrics.counter("agent.skill.chosen", "skill" to name).increment()
}

Debugging routing decisions:

onSkillChosen { name ->
    logger.debug("Skill '$name' chosen for input: ${currentInput.take(100)}...")
}

onError

Fires when an infrastructure error is about to leave an agentic invocation: LLM transport failures, response parsing failures, budget exceptions, skill-routing failures, and similar runtime failures.

agent.onError { error ->
    logger.warn("agent failed: ${error::class.simpleName}: ${error.message}")
}

This is not tool recovery. Per-tool recovery still lives in Tool Error Recovery; onError observes and then the original exception propagates.

onBudgetThreshold

Fires once per cumulative BudgetReason when usage crosses the configured threshold. It is useful for warning, logging, or graceful wrap-up before a cap throws.

agent<String, String>("researcher") {
    budget {
        maxTurns = 10
        maxToolCalls = 20
        maxTokens = 8_000
    }

    onBudgetThreshold(0.8) { reason, usedPercent ->
        logger.warn("budget warning: $reason at ${(usedPercent * 100).toInt()}%")
    }
}

TOKENS warnings require budget.maxTokens and provider-reported token usage.

observe

Use observe { } when telemetry code prefers one sealed event stream instead of separate listener registrations.

agent.observe { event ->
    when (event) {
        is PipelineEvent.SkillChosen -> logger.info("skill=${event.skillName}")
        is PipelineEvent.ToolCalled -> logger.info("tool=${event.toolName}")
        is PipelineEvent.KnowledgeLoaded -> logger.info("knowledge=${event.entryName}")
        is PipelineEvent.ErrorOccurred -> logger.warn("error=${event.error.message}")
    }
}

Multiple observe { } calls stack additively. Existing onSkillChosen, onToolUse, onKnowledgeUsed, and onError listeners fire first, then the unified observer receives its event.

Combining Hooks

An agent can combine the focused hooks when separate callback slots are easier than a unified observe { } stream:

val agent = agent<String, String>("observable-agent") {
    model { ollama("qwen2.5:7b") }
    budget { maxTurns = 10 }

    lateinit var search: Tool<Map<String, Any?>, Any?>
    lateinit var fetchPage: Tool<Map<String, Any?>, Any?>

    tools {
        search = tool("search", "Web search") { args ->
            webSearch(args["query"] as String)
        }
        fetchPage = tool("fetch_page", "Fetch a web page") { args ->
            httpClient.get(args["url"] as String)
        }
    }

    skills {
        skill<String, String>("research", "Research a topic") {
            tools(search, fetchPage)

            knowledge("guidelines", "Research guidelines") {
                loadText("/data/guidelines.txt")
            }
        }

        skill<String, String>("summarize", "Summarize content") {
            implementedBy { input -> summarize(input) }
        }
    }

    onSkillChosen { name ->
        logger.info("skill_selected: $name")
    }

    onKnowledgeUsed { name, content ->
        logger.info("knowledge_fetched: $name (${content.length} chars)")
    }

    onToolUse { name, args, result ->
        logger.info("tool_executed: $name args=$args")
    }
}

A typical execution trace:

skill_selected: research
knowledge_fetched: guidelines (1205 chars)
tool_executed: search args={query=Kotlin coroutines best practices}
tool_executed: fetch_page args={url=https://example.com/article}

The hooks fire in execution order: skill selection first, then knowledge and tools as the agentic loop runs.

Testing with Hooks

Hooks are powerful testing tools. They let you assert on agent behavior without needing a live LLM.

Assert on Tool Calls

@Test
fun `agent calls search before summarize`() {
    val toolCalls = mutableListOf<String>()

    val mockClient = ModelClient { messages ->
        when (toolCalls.size) {
            0 -> LlmResponse.ToolCalls(listOf(ToolCall("search", mapOf("query" to "test"))))
            1 -> LlmResponse.ToolCalls(listOf(ToolCall("summarize", mapOf("text" to "data"))))
            else -> LlmResponse.Text("Summary: test data")
        }
    }

    val agent = agent<String, String>("test-agent") {
        model { ollama("unused"); client = mockClient }
        budget { maxTurns = 5 }

        lateinit var search: Tool<Map<String, Any?>, Any?>
        lateinit var summarize: Tool<Map<String, Any?>, Any?>
        tools {
            search    = tool("search",    "Search")    { "results" }
            summarize = tool("summarize", "Summarize") { "summary" }
        }

        skills {
            skill<String, String>("work", "Do work") {
                tools(search, summarize)
            }
        }

        onToolUse { name, _, _ ->
            toolCalls.add(name)
        }
    }

    agent("analyze this")

    assertEquals(listOf("search", "summarize"), toolCalls)
}

Assert on Skill Selection

@Test
fun `billing questions route to billing skill`() {
    var selectedSkill = ""

    val agent = agent<String, String>("router") {
        skillSelection { input ->
            if (input.contains("charge")) "billing" else "general"
        }

        skills {
            skill<String, String>("billing", "Billing") {
                implementedBy { "Billing response" }
            }
            skill<String, String>("general", "General") {
                implementedBy { "General response" }
            }
        }

        onSkillChosen { name ->
            selectedSkill = name
        }
    }

    agent("Why was I charged twice?")

    assertEquals("billing", selectedSkill)
}

Assert on Knowledge Access

@Test
fun `agent uses FAQ knowledge for common questions`() {
    val knowledgeAccessed = mutableListOf<String>()

    val mockClient = ModelClient { messages ->
        // Simulate the LLM fetching knowledge, then answering
        if (knowledgeAccessed.isEmpty()) {
            LlmResponse.ToolCalls(listOf(ToolCall("knowledge_faq", emptyMap())))
        } else {
            LlmResponse.Text("Based on the FAQ: yes, we offer refunds.")
        }
    }

    val agent = agent<String, String>("support") {
        model { ollama("unused"); client = mockClient }
        budget { maxTurns = 3 }

        skills {
            skill<String, String>("answer", "Answer questions") {
                tools("knowledge_faq") // built-in / framework-exposed knowledge tool — no user handle

                knowledge("faq", "FAQ content") {
                    "Q: Do you offer refunds? A: Yes, within 30 days."
                }
            }
        }

        onKnowledgeUsed { name, _ ->
            knowledgeAccessed.add(name)
        }
    }

    agent("Do you offer refunds?")

    assertTrue(knowledgeAccessed.contains("faq"))
}

Full Integration Test Pattern

Combine all hooks for comprehensive behavior verification:

@Test
fun `full agent behavior test`() {
    var skill = ""
    val tools = mutableListOf<String>()
    val knowledge = mutableListOf<String>()

    val mockClient = ModelClient { messages ->
        when (tools.size) {
            0 -> LlmResponse.ToolCalls(listOf(ToolCall("search", mapOf("q" to "test"))))
            else -> LlmResponse.Text("answer")
        }
    }

    val agent = agent<String, String>("full-test") {
        model { ollama("unused"); client = mockClient }
        budget { maxTurns = 5 }

        lateinit var search: Tool<Map<String, Any?>, Any?>
        tools {
            search = tool("search", "Search") { "results" }
        }

        skills {
            skill<String, String>("research", "Research") {
                tools(search)

                knowledge("docs", "Documentation") { "doc content" }
            }
        }

        onSkillChosen { skill = it }
        onToolUse { name, _, _ -> tools.add(name) }
        onKnowledgeUsed { name, _ -> knowledge.add(name) }
    }

    val result = agent("find info")

    assertEquals("research", skill)
    assertEquals(listOf("search"), tools)
    assertEquals("answer", result)
}

This pattern gives you deterministic, fast, LLM-free tests that verify the agent's orchestration logic: which skill was chosen, which tools were called, in what order, and what the final result was.

Next Steps

Agents.KT Wiki

Project Links

Getting Started

Core Concepts

Composition Operators

LLM Integration

Guided Generation

Agent Memory

Reference

Contributing

Clone this wiki locally