Skip to content

Commit 131c72e

Browse files
Merge pull request #32 from Factory-AI/ainesh/mb-17-callout-overlay
2 parents e8801fa + 5654245 commit 131c72e

10 files changed

Lines changed: 192 additions & 18 deletions

File tree

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
.factory/
2+
node_modules/

plugins/droid-control/ARCHITECTURE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ Terminal workflows use `bin/tctl` as the only launch/control boundary. It hides
117117
| `tuistory` | Virtual PTY sessions, deterministic waits/snapshots, asciinema recording at launch. | Fast TUI automation and most demo captures. |
118118
| `true-input` | Headless Wayland compositor, real terminal emulator, native key injection, PTY log/screenshot/video capture. | Real terminal rendering or keyboard-encoding proof. |
119119

120-
`tctl` also enforces Droid CLI launch invariants. `droid-dev` sessions must provide `--repo-root`, which lets `tctl` set `DROID_DEV_REPO_ROOT` and record provenance for the captured branch and commit.
120+
`tctl` also enforces Droid CLI launch invariants. `droid-dev` sessions must provide `--repo-root`, which lets `tctl` set `DROID_DEV_REPO_ROOT` and record provenance for the captured branch and commit. `--cwd` is independent: it sets the child's working directory (recorded in provenance) and defaults to `--repo-root` when unset, so a session can run one worktree's code from a different project directory.
121121

122122
Browser/Electron and native-desktop workflows intentionally do **not** go through `tctl`. They have their own control boundaries: `agent-browser`'s persistent Playwright daemon for DOM snapshots, screenshots, and CDP-connected apps; `cua-driver`'s daemon for accessibility trees and per-`(pid, window_id)` element caches on desktop GUIs.
123123

plugins/droid-control/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ For the full rationale and runtime pipeline, see [`ARCHITECTURE.md`](ARCHITECTUR
7676

7777
## Video rendering
7878

79-
The compose stage uses [Remotion](https://www.remotion.dev/) for video compositing. Presets provide window chrome, spacing, palettes, backgrounds, particles, noise, color grading, configurable transitions (`motion-blur`, `flash`, `whip-pan`, `light-leak`, `glitch-lite`), zooms, spotlights, keystroke overlays, section headers, and syntax-highlighted code annotations.
79+
The compose stage uses [Remotion](https://www.remotion.dev/) for video compositing. Presets provide window chrome, spacing, palettes, backgrounds, particles, noise, color grading, configurable transitions (`motion-blur`, `flash`, `whip-pan`, `light-leak`, `glitch-lite`), zooms, spotlights, callout annotations, keystroke overlays, section headers, and syntax-highlighted code annotations.
8080

8181
The `render-showcase.sh` helper owns the full pipeline: `.cast` conversion via `agg`, clip staging, duration detection, Remotion rendering, and cleanup.
8282

plugins/droid-control/bin/tctl

Lines changed: 21 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ Launch options:
3131
--terminal <name> Override terminal for true-input
3232
--cols <n> Columns (default: 120)
3333
--rows <n> Rows (default: 36)
34-
--cwd <path> Working directory (tuistory)
34+
--cwd <path> Working directory for the child, both backends
35+
(default: --repo-root when set)
3536
--repo-root <path> Git worktree root for provenance + droid-dev launches
3637
--env <KEY=VALUE> Environment variable (repeatable)
3738
--tmux Wrap the command in tmux with RGB-safe tuistory defaults
@@ -319,21 +320,25 @@ require_git_worktree_root() {
319320
write_provenance() {
320321
local session="$1"
321322
local repo_root="$2"
323+
local cwd="$3"
322324
local file branch commit
323325
file="$(provenance_file "$session")"
324326

325-
if [[ -z "$repo_root" ]]; then
327+
if [[ -z "$repo_root" && -z "$cwd" ]]; then
326328
rm -f "$file"
327329
return 0
328330
fi
329331

330-
branch="$(git -C "$repo_root" rev-parse --abbrev-ref HEAD 2>/dev/null || printf 'unknown')"
331-
commit="$(git -C "$repo_root" rev-parse HEAD 2>/dev/null || printf 'unknown')"
332-
cat > "$file" <<PROVENANCE
333-
repo_root=$repo_root
334-
branch=$branch
335-
commit=$commit
336-
PROVENANCE
332+
{
333+
if [[ -n "$repo_root" ]]; then
334+
branch="$(git -C "$repo_root" rev-parse --abbrev-ref HEAD 2>/dev/null || printf 'unknown')"
335+
commit="$(git -C "$repo_root" rev-parse HEAD 2>/dev/null || printf 'unknown')"
336+
printf 'repo_root=%s\nbranch=%s\ncommit=%s\n' "$repo_root" "$branch" "$commit"
337+
fi
338+
if [[ -n "$cwd" ]]; then
339+
printf 'cwd=%s\n' "$cwd"
340+
fi
341+
} > "$file"
337342
}
338343

339344
resolve_backend() {
@@ -937,9 +942,12 @@ cmd_launch() {
937942

938943
if [[ -n "$repo_root" ]]; then
939944
require_git_worktree_root "$repo_root"
940-
if [[ -n "$cwd" && "$cwd" != "$repo_root" ]]; then
941-
die "--cwd must match --repo-root when --repo-root is set"
942-
fi
945+
fi
946+
947+
if [[ -n "$cwd" ]]; then
948+
[[ "$cwd" = /* ]] || die "--cwd must be an absolute path: $cwd"
949+
[[ -d "$cwd" ]] || die "--cwd does not exist: $cwd"
950+
elif [[ -n "$repo_root" ]]; then
943951
cwd="$repo_root"
944952
fi
945953

@@ -977,7 +985,7 @@ cmd_launch() {
977985
WARMED_UP="0"
978986
REPO_ROOT="$repo_root"
979987
save_session_state "$session"
980-
write_provenance "$session" "$repo_root"
988+
write_provenance "$session" "$repo_root" "$cwd"
981989

982990
if [[ "$BACKEND" == "tuistory" ]]; then
983991
launch_tuistory "$session" "$cols" "$rows" "$record_path"
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
import { useCurrentFrame, useVideoConfig, interpolate, Easing } from 'remotion';
2+
import type { Effect } from '../lib/schema';
3+
import type { Palette } from '../lib/palettes';
4+
5+
type Callout = Extract<Effect, { fx: 'callout' }>;
6+
7+
const CalloutPill: React.FC<{
8+
callout: Callout;
9+
palette: Palette;
10+
enterFrame: number;
11+
exitFrame: number;
12+
}> = ({ callout, palette, enterFrame, exitFrame }) => {
13+
const frame = useCurrentFrame();
14+
const { fps } = useVideoConfig();
15+
16+
const isVisible = frame >= enterFrame && frame < exitFrame;
17+
if (!isVisible) return null;
18+
19+
const localFrame = frame - enterFrame;
20+
21+
// Pop-in animation over 0.25s (same overshoot bezier as KeystrokePill)
22+
const enterProgress = interpolate(localFrame, [0, 0.25 * fps], [0, 1], {
23+
easing: Easing.bezier(0.34, 1.56, 0.64, 1),
24+
extrapolateLeft: 'clamp',
25+
extrapolateRight: 'clamp',
26+
});
27+
28+
// Fade out over last 0.2s
29+
const totalFrames = exitFrame - enterFrame;
30+
const fadeOutStart = totalFrames - 0.2 * fps;
31+
const exitOpacity = interpolate(
32+
localFrame,
33+
[fadeOutStart, totalFrames],
34+
[1, 0],
35+
{
36+
extrapolateLeft: 'clamp',
37+
extrapolateRight: 'clamp',
38+
}
39+
);
40+
41+
const scale = interpolate(enterProgress, [0, 1], [0.85, 1]);
42+
43+
return (
44+
<div
45+
style={{
46+
position: 'absolute',
47+
left: callout.at.x,
48+
top: callout.at.y,
49+
transform: `translate(-50%, -50%) scale(${scale})`,
50+
opacity: Math.min(enterProgress, exitOpacity),
51+
backgroundColor: `${palette.surface}E6`,
52+
color: palette.text,
53+
fontSize: 24,
54+
fontFamily: "'Geist', 'Inter', sans-serif",
55+
fontWeight: 500,
56+
lineHeight: 1.4,
57+
padding: '12px 24px',
58+
borderRadius: 12,
59+
border: `1px solid ${palette.border}`,
60+
borderLeft: `3px solid ${palette.accent}`,
61+
backdropFilter: 'blur(12px)',
62+
boxShadow: '0 8px 32px rgba(0, 0, 0, 0.35)',
63+
maxWidth: '40%',
64+
textAlign: 'center',
65+
zIndex: 90,
66+
pointerEvents: 'none',
67+
}}
68+
>
69+
{callout.text}
70+
</div>
71+
);
72+
};
73+
74+
export const CalloutOverlay: React.FC<{
75+
callouts: Callout[];
76+
palette: Palette;
77+
}> = ({ callouts, palette }) => {
78+
const { fps } = useVideoConfig();
79+
80+
return (
81+
<>
82+
{callouts.map((callout) => (
83+
<CalloutPill
84+
key={`${callout.t}-${callout.text}`}
85+
callout={callout}
86+
palette={palette}
87+
enterFrame={Math.round(callout.t * fps)}
88+
exitFrame={Math.round((callout.t + callout.dur) * fps)}
89+
/>
90+
))}
91+
</>
92+
);
93+
};

plugins/droid-control/remotion/src/compositions/Showcase.tsx

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ import { ZoomEffect } from '../components/ZoomEffect';
3131
import { SectionTransitionOverlay } from '../components/SectionTransition';
3232
import { DroidOutro } from '../components/DroidOutro';
3333
import { CodeAnnotationOverlay } from '../components/CodeAnnotationOverlay';
34+
import { CalloutOverlay } from '../components/CalloutOverlay';
3435

3536
export const showcaseSchema = z.object({
3637
clips: z.array(z.string()),
@@ -68,6 +69,23 @@ export const showcaseSchema = z.object({
6869
const TITLE_DURATION_S = 4;
6970
const TRANSITION_FRAMES = 15;
7071

72+
// Effect types this composition actually renders. Anything schema-valid but
73+
// absent from this set is a silent no-op — warn instead of dropping quietly.
74+
const RENDERED_FX = new Set(['zoom', 'spotlight', 'callout']);
75+
const warnedUnrenderedFx = new Set<string>();
76+
77+
const warnUnrenderedEffects = (effects: z.infer<typeof effectSchema>[]) => {
78+
for (const effect of effects) {
79+
if (RENDERED_FX.has(effect.fx) || warnedUnrenderedFx.has(effect.fx)) {
80+
continue;
81+
}
82+
warnedUnrenderedFx.add(effect.fx);
83+
console.warn(
84+
`Showcase: effect fx='${effect.fx}' is schema-valid but not rendered by this composition`
85+
);
86+
}
87+
};
88+
7189
const resolveFidelity = (
7290
props: z.infer<typeof showcaseSchema>
7391
): 'compact' | 'standard' | 'inspect' =>
@@ -211,6 +229,16 @@ export const ShowcaseComposition: React.FC<z.infer<typeof showcaseSchema>> = (
211229
[props.effects]
212230
);
213231

232+
const callouts = useMemo(
233+
() =>
234+
props.effects.filter(
235+
(e): e is Extract<typeof e, { fx: 'callout' }> => e.fx === 'callout'
236+
),
237+
[props.effects]
238+
);
239+
240+
warnUnrenderedEffects(props.effects);
241+
214242
return (
215243
<AbsoluteFill>
216244
<Background palette={palette} config={config} />
@@ -289,6 +317,11 @@ export const ShowcaseComposition: React.FC<z.infer<typeof showcaseSchema>> = (
289317
/>
290318
))}
291319

320+
{/* Callout annotations: timed text pills at percent positions */}
321+
{callouts.length > 0 && (
322+
<CalloutOverlay callouts={callouts} palette={palette} />
323+
)}
324+
292325
{/* Frosted sweep at section boundaries */}
293326
{props.sections && props.sections.length > 1 && (
294327
<SectionTransitionOverlay sections={props.sections} />

plugins/droid-control/skills/compose/SKILL.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -245,8 +245,8 @@ Use a run-scoped props path like `$PROPS`; do not reuse a global `/tmp/showcase-
245245

246246
| Effect | Props | Description |
247247
|---|---|---|
248-
| `fade-in` | `t`, `dur` | Fade from black |
249-
| `fade-out` | `t`, `dur` | Fade to black |
248+
| `fade-in` | `t`, `dur` | Fade from black (schema-valid but not yet rendered by Showcase — emits a render-time warning) |
249+
| `fade-out` | `t`, `dur` | Fade to black (schema-valid but not yet rendered by Showcase — emits a render-time warning) |
250250
| `zoom` | `t`, `dur`, `to: {x,y,w,h}` | Directed zoom to a target region (30% in, 40% hold, 30% out) |
251251
| `spotlight` | `t`, `dur`, `on: {x,y,w,h}`, `dim?` | Dim everything except a region (`dim`: 0–1, default 0.6) |
252252
| `callout` | `t`, `dur`, `text`, `at: {x,y}` | Text overlay anchored to a point |

plugins/droid-control/skills/droid-cli/SKILL.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ $TCTL -s demo press tab
7272
- **No per-branch builds.** One `npm run setup` (in any checkout) installs the shim. Switching branches is instant via `--repo-root`.
7373
- **Prerequisite:** The target worktree must have `node_modules` installed (`npm install` at the repo root). If missing, the bun launch fails.
7474
- **`tctl --repo-root`** sets `DROID_DEV_REPO_ROOT` automatically and pins the session to that worktree.
75+
- **`tctl --cwd`** is independent of `--repo-root`: run droid against one worktree's code while sitting in a different project directory (e.g., a scratch project with its own `.factory/settings.json` hooks). Defaults to `--repo-root` when unset.
7576

7677
### `droid-dev`
7778

plugins/droid-control/skills/droid-control/SKILL.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,44 @@ Terminal drivers use the unified `tctl` wrapper. agent-browser and desktop-contr
162162

163163
Drivers can be combined in one workflow — e.g., `tctl` for a CLI and `agent-browser` for a web UI it interacts with.
164164

165+
## Degraded-tail repro recipe (droid TUI)
166+
167+
Deterministic recipe for reproducing degraded transcript tails in the droid CLI — stranded live tool rows and queued steering messages — without waiting for a slow model turn. The trick: a slow `PreToolUse` hook pins a tool in its executing state for as long as you need.
168+
169+
1. **Scratch project.** Create a throwaway directory (never a real repo — the hook fires on every matching tool call) with a project-local hook that sleeps:
170+
171+
```bash
172+
SCRATCH="$(mktemp -d /tmp/degraded-tail-XXXXXX)"
173+
mkdir -p "$SCRATCH/.factory"
174+
cat > "$SCRATCH/.factory/settings.json" <<'JSON'
175+
{
176+
"hooks": {
177+
"PreToolUse": [
178+
{
179+
"matcher": "TodoWrite",
180+
"hooks": [{ "type": "command", "command": "sleep 120" }]
181+
}
182+
]
183+
}
184+
}
185+
JSON
186+
```
187+
188+
Pick a sleep long enough to interact mid-hook (60–180s) and a matcher for a tool the prompt will reliably trigger (`TodoWrite` fires on any multi-step ask).
189+
190+
2. **Launch with `--cwd` pointed at the scratch project** — `--repo-root` stays on your dev worktree so `droid-dev` provenance still records the code under test:
191+
192+
```bash
193+
$TCTL launch "droid-dev" -s ${RUN_ID}-tail --cwd "$SCRATCH" \
194+
--repo-root /abs/path/to/dev/worktree --record ${RUN_DIR}/tail.cast
195+
```
196+
197+
3. **Trigger the hook**, then degrade the tail while the tool row shows executing:
198+
- **Interrupt mid-hook** (`press escape`) — strands the live tool row: it never resolves to a completed/canceled state in the transcript tail.
199+
- **Steer mid-hook** (`type "..."` + `press enter`) — the steering message queues behind the executing tool instead of interleaving.
200+
201+
**Gotcha:** dev-scope hook settings can silently disable project hooks. If the tool completes instantly, run `/hooks` in the session and check the "Hooks enabled" toggle before debugging the hook config itself.
202+
165203
## Prerequisites
166204
167205
| Stage | Platform | Required | Optional |

plugins/droid-control/skills/tuistory/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ $TCTL -s demo close
5959
| `snapshot [--trim]` | Print cleaned text (`--trim` strips trailing blanks) |
6060
| `close` | Tear down session |
6161

62-
Launch options: `--cols <n>`, `--rows <n>`, `--cwd <path>`, `--env KEY=VALUE`, `--record <path>`.
62+
Launch options: `--cols <n>`, `--rows <n>`, `--cwd <path>` (child working directory; defaults to `--repo-root` when set), `--env KEY=VALUE`, `--record <path>`.
6363

6464
## Recording
6565

0 commit comments

Comments
 (0)