Feature Request: AI agent skill for serverless review

[floatious]

If we look at Chris Mason's review-prompts git tree (which is synced into the sashiko git tree periodically),
it has a /kseries skill that can be invoked using /kseries within an AI agent, to perform a review:
https://github.com/masoncl/review-prompts/blob/main/kernel/skills/kernel.md
https://github.com/masoncl/review-prompts/blob/main/kernel/review-core.md

However, while the review done by /kseries has 6 stages (tasks 0-5), the review done by Sashiko has 10 stages
(where stages 3,5,8,9 are actually defined in Chris Mason's review-prompts).

Thus, the review done by Sashiko is not identical to the review done by /kseries.

Being able to simply invoke a skill from within the AI agent, avoids the need to have a server running
(and even avoids the need to compile any binary, or to have cargo installed).

It would thus be nice if Sashiko could implement a skill similar to /kseries, to facilitate serverless reviewing.
This would basically make a serverless standalone binary as proposed in: #134 unnecessary.

It seems like skills are more or less compatible between agents, so while e.g. Chris Mason's review-prompts
git tree allows you to install to different paths (e.g. .claude/skills / .codex/.agents/skills / .opencode/skills ):
https://github.com/masoncl/review-prompts/blob/main/setup.sh

The actual files setup.sh installs are the same, and many AI agents, e.g. devin for terminal, will automatically
load skills installed for other AI agents.

[derekbarbosa]

Hey @floatious ! Thanks for the issue and detailed description.

So this one is a weird one. Not for any technical reason, but mainly because the discussion of "where should these prompts live" (wrt to mason's prompts) is one that is still happening. (@rgushchin correct me if I am wrong).

Not that it would be "difficult" to maintain a set of skills that correspond to the prompts.rs file, but I can see it being another thing to manage in terms of drift. I'm hoping #188 resolves this.

Anyway, I'm pushing up a branch with the desired agent skills in a minute. It doesn't have an install script, so just point your preferred agent at the skills/ directory.

Let me know if it works out well for you! :)

[derekbarbosa]

https://github.com/derekbarbosa/sashiko-dev/tree/feature/agent-skill

[rgushchin]

It's a tricky one:

1. 3rd-party agentic coding tools ingest custom system prompts. They do provide a different and potentially richer set of tools. They do some preprocessing of results. They might make automatic choices which model to use.
2. One of the main points of Sashiko was to make the entire process as open-source and controllable by the kernel community as possible (within current realistic limitations).
3. Sashiko was based on Chris Mason's work - which is exactly that, a set of prompts/skills to use with agentic coding tools like claude code. In a way, there is no point in going back, we can simple use his setup.
4. I totally get the point that having something like this might be convenient during the development process.

So summarizing all above I'd say I'm not against putting something like this into the Sashiko's repository, given that we make it very clear that results are not guaranteed to match the "official" sashiko review and will likely be sufficiently different (both now and going forward). Maybe we should call it pre-review or something like this.

[floatious]

Hello @derekbarbosa @rgushchin ,

The use case that I want to cover is indeed for a developer to be able to run an AI assisted review before sending out a series,
without having a whole Sashiko server running.

In that way, calling it pre-review sounds like a good idea.

I assume that people want it to yield as similar review comments as possible to the "official" Sashiko review, such that it is
possible to fix as many comments as possible before, rather than after, sending out a patch series.

If you think that the suggestion in #134 would yield more similar review comments compared the suggestion in this issue, then perhaps #134 is a better approach? Unless you think there is value in having both?

I am not an LLM expert, so I don't see why we would get different results, assuming that, like Derek suggested, the agent writes out the findings to a file between stages. I though that Sashiko's output was already probabilistic: it might find or not find bugs (or find other bugs) with the same input. But perhaps having an AI agent skill would still be "more probabilistic",
compared e.g. the approach in #134 ?

With regards to being controllable by the kernel community, I know that Derek is adding a custom Stage Configuration via a stages.toml file in #188, to control the review pipeline (enable or disable individual stages). But AFAICT, an AI agent skill can also load a .toml file, so I assumed that even an AI agent skill would still be quite "controllable".

[derekbarbosa]

Hi @floatious,

Yes, I understand your use case, it is a perfectly reasonable one. I can change the name of the skill to reflect /pre-review.

I am unsure whether #134 or this proposed agent skill would differ in review output. With the removal of TodoWrite, the difference (at a glance) between the serverless-binary becomes worktrees and "stages" with propagation of JSON between them. I haven't run any benchmarks to determine what provides a "consistent" review between these two yet.

[TheSven73]

so I don't see why we would get different results

I suspect you might actually get very different results with a /patch-review skill, and not in a good way. I'm no AI or Sashiko expert, but here is what I suspect:

LLMs move from the "Smart Zone" to the "Dumb Zone" when their context gets too large. Sashiko tries to keep its LLM in the Smart Zone by starting over with a fresh, condensed, relevant context at the start of each Stage.

A Skill or Slash Command does not support context clearing. It is much more likely that the LLM's context will grow pretty large as it moves through the stages within a Skill. Depending on the model you use, it will eventually enter the Dumb Zone and/or become generally less effective.

One possible way to replicate Sashiko's behavior with just Claude Code or Antigravity (and without #134) is to use a Ralph Loop. You would need a well-formulated Ralph Prompt that instructs the LLM to move through the stages like a state machine and maintain its current state in a Markdown file. This basically asks the LLM to take over the context trimming and summarization functionality from Sashiko. It's unknown how well that would work in practice, so your mileage may vary.

[rgushchin]

My 2c here:

1. I'm about to merge a big change into sashiko which will accelerate reviews 2-3x, so it will be more convenient for the developer's usage.
2. These things are super sensitive to even minor changes in the harness. E.g. you sort grep output in a different way and it starts finding some new bugs, for example. 3rd party coding tools do include their own prompts and they have different tools implementation, so we never get the same out of it.
3. Instead of trying to get similar results, I suggest to do the opposite: introduce a way to conveniently export Sashiko's review context to CLI tools. This will allow developers to continue the session interactively but based on full review context.