Skip to content

Commit 6c94c1c

Browse files
docs: add smooth mouse movement and drag documentation (#251)
Document the smooth Bezier curve movement for moveMouse and dragMouse, including parameter tables, code examples in TS/Python/CLI, and demo GIFs showing smooth vs instant/linear movement. Made-with: Cursor
1 parent facf1ed commit 6c94c1c

3 files changed

Lines changed: 147 additions & 19 deletions

File tree

browsers/computer-controls.mdx

Lines changed: 147 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,29 @@ description: "Control the computer's mouse, keyboard, and screen"
55

66
Use OS-level controls to move and click the mouse, type and press keys, scroll, drag, and capture screenshots from a running browser session.
77

8+
<Tip>
9+
Smooth movement is enabled by default — every `moveMouse` and `dragMouse` call already uses [Bézier curves](https://en.wikipedia.org/wiki/B%C3%A9zier_curve). Run this script and open the [live view](/browsers/live-view) link to watch it in action:
10+
11+
```python smooth_demo.py
12+
from kernel import Kernel
13+
import time, urllib.request
14+
15+
k = Kernel()
16+
b = k.browsers.create()
17+
print(f"\n Watch live: {b.live_view_url}\n")
18+
19+
html = urllib.request.urlopen(
20+
"https://gist.githubusercontent.com/ulziibay-kernel/f6d8063ba223a81293cfe1fde06d8624/raw/cursor-trail-demo.html"
21+
).read().decode()
22+
k.browsers.playwright.execute(id=b.session_id, code="await page.setContent(" + repr(html) + ");")
23+
input("Press Enter when live view is open...")
24+
25+
for x, y in [(200, 200), (900, 150), (1700, 250), (1700, 700), (960, 800), (200, 750), (500, 480), (1400, 480)]:
26+
k.browsers.computer.move_mouse(id=b.session_id, x=x, y=y, duration_ms=1200)
27+
time.sleep(0.3)
28+
```
29+
</Tip>
30+
831
## Click the mouse
932

1033
Simulate mouse clicks at specific coordinates. You can select the button, click type (down, up, click), number of clicks, and optional modifier keys to hold.
@@ -69,7 +92,15 @@ kernel browsers computer click-mouse <session id> --x 100 --y 200 --num-clicks 2
6992

7093
## Move the mouse
7194

72-
Move the cursor to specific screen coordinates. Optionally hold modifier keys during the move.
95+
Move the cursor to specific screen coordinates. By default, the cursor follows a human-like Bezier curve path instead of teleporting instantly. You can control this with `smooth` and `duration_ms`.
96+
97+
| Parameter | Type | Default | Description |
98+
| ------------- | ------- | ------- | ----------- |
99+
| `x` | integer || X coordinate to move the cursor to |
100+
| `y` | integer || Y coordinate to move the cursor to |
101+
| `smooth` | boolean | `true` | Use human-like Bezier curve path instead of instant teleport |
102+
| `duration_ms` | integer | auto | Target duration in milliseconds for smooth movement (50–5000). Omit for automatic timing based on distance |
103+
| `hold_keys` | array || Modifier keys to hold during the move |
73104

74105
<CodeGroup>
75106
```typescript Typescript/Javascript
@@ -78,10 +109,25 @@ import Kernel from '@onkernel/sdk';
78109
const kernel = new Kernel();
79110
const kernelBrowser = await kernel.browsers.create();
80111

112+
// Human-like smooth movement (default)
81113
await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
82114
x: 500,
83115
y: 300,
84-
hold_keys: ['Alt'],
116+
});
117+
118+
// Smooth movement with custom duration
119+
await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
120+
x: 800,
121+
y: 600,
122+
smooth: true,
123+
duration_ms: 1500,
124+
});
125+
126+
// Instant teleport (disable smooth)
127+
await kernel.browsers.computer.moveMouse(kernelBrowser.session_id, {
128+
x: 100,
129+
y: 200,
130+
smooth: false,
85131
});
86132
```
87133

@@ -91,20 +137,46 @@ from kernel import Kernel
91137
kernel = Kernel()
92138
kernel_browser = kernel.browsers.create()
93139

140+
# Human-like smooth movement (default)
94141
kernel.browsers.computer.move_mouse(
95142
id=kernel_browser.session_id,
96143
x=500,
97144
y=300,
98-
hold_keys=["Alt"],
145+
)
146+
147+
# Smooth movement with custom duration
148+
kernel.browsers.computer.move_mouse(
149+
id=kernel_browser.session_id,
150+
x=800,
151+
y=600,
152+
smooth=True,
153+
duration_ms=1500,
154+
)
155+
156+
# Instant teleport (disable smooth)
157+
kernel.browsers.computer.move_mouse(
158+
id=kernel_browser.session_id,
159+
x=100,
160+
y=200,
161+
smooth=False,
99162
)
100163
```
101164

102165
```bash CLI
103-
# Move the mouse to coordinates (500, 300)
166+
# Smooth movement (default)
104167
kernel browsers computer move-mouse <session id> --x 500 --y 300
168+
169+
# Instant teleport
170+
kernel browsers computer move-mouse <session id> --x 500 --y 300 --smooth=false
105171
```
106172
</CodeGroup>
107173

174+
### Smooth vs instant movement
175+
176+
<Frame caption="Blue = smooth Bezier path, Red = instant teleport">
177+
<img src="/images/smooth-mouse-demo.gif" />
178+
</Frame>
179+
108180
## Take screenshots
109181

110182
Capture a full-screen PNG or a specific region.
@@ -310,7 +382,18 @@ kernel browsers computer scroll <session id> --x 300 --y 400 --delta-y 120
310382

311383
## Drag the mouse
312384

313-
Drag by pressing a button, moving along a path of points, then releasing. You can control delay before starting, the granularity and speed of the drag via `steps_per_segment` and `step_delay_ms`, and optionally hold modifier keys.
385+
Drag by pressing a button, moving along a path of points, then releasing. By default, drag movement uses human-like Bezier curves between waypoints. Set `smooth: false` to use linear interpolation with `steps_per_segment` and `step_delay_ms` instead.
386+
387+
| Parameter | Type | Default | Description |
388+
| ------------------- | ------- | ------- | ----------- |
389+
| `path` | array || Ordered list of `[x, y]` coordinate pairs to move through (minimum 2 points) |
390+
| `button` | string | `left` | Mouse button: `left`, `middle`, or `right` |
391+
| `smooth` | boolean | `true` | Use human-like Bezier curves between waypoints. When `true`, `steps_per_segment` and `step_delay_ms` are ignored |
392+
| `duration_ms` | integer | auto | Target duration in milliseconds for the entire drag when `smooth=true` (50–10000). Omit for automatic timing based on total path length |
393+
| `delay` | integer | `0` | Delay in milliseconds between button down and starting to move |
394+
| `steps_per_segment` | integer | `10` | Number of interpolation steps per path segment (only when `smooth=false`) |
395+
| `step_delay_ms` | integer | `50` | Delay in milliseconds between steps (only when `smooth=false`) |
396+
| `hold_keys` | array || Modifier keys to hold during the drag |
314397

315398
<CodeGroup>
316399
```typescript Typescript/Javascript
@@ -319,17 +402,36 @@ import Kernel from '@onkernel/sdk';
319402
const kernel = new Kernel();
320403
const kernelBrowser = await kernel.browsers.create();
321404

405+
// Human-like smooth drag (default)
406+
await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
407+
path: [
408+
[100, 200],
409+
[400, 350],
410+
[700, 200],
411+
],
412+
});
413+
414+
// Smooth drag with custom duration
322415
await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
323416
path: [
324417
[100, 200],
325-
[150, 220],
326-
[200, 260],
418+
[400, 350],
419+
[700, 200],
327420
],
328-
button: 'left',
329-
delay: 0,
421+
smooth: true,
422+
duration_ms: 2000,
423+
});
424+
425+
// Linear interpolation drag (legacy behavior)
426+
await kernel.browsers.computer.dragMouse(kernelBrowser.session_id, {
427+
path: [
428+
[100, 200],
429+
[400, 350],
430+
[700, 200],
431+
],
432+
smooth: false,
330433
steps_per_segment: 10,
331434
step_delay_ms: 50,
332-
hold_keys: ['Shift'],
333435
});
334436
```
335437

@@ -339,24 +441,50 @@ from kernel import Kernel
339441
kernel = Kernel()
340442
kernel_browser = kernel.browsers.create()
341443

444+
# Human-like smooth drag (default)
342445
kernel.browsers.computer.drag_mouse(
343446
id=kernel_browser.session_id,
344-
path=[[100, 200], [150, 220], [200, 260]],
345-
button="left",
346-
delay=0,
447+
path=[[100, 200], [400, 350], [700, 200]],
448+
)
449+
450+
# Smooth drag with custom duration
451+
kernel.browsers.computer.drag_mouse(
452+
id=kernel_browser.session_id,
453+
path=[[100, 200], [400, 350], [700, 200]],
454+
smooth=True,
455+
duration_ms=2000,
456+
)
457+
458+
# Linear interpolation drag (legacy behavior)
459+
kernel.browsers.computer.drag_mouse(
460+
id=kernel_browser.session_id,
461+
path=[[100, 200], [400, 350], [700, 200]],
462+
smooth=False,
347463
steps_per_segment=10,
348464
step_delay_ms=50,
349-
hold_keys=["Shift"],
350465
)
351466
```
352467

353468
```bash CLI
354-
# Drag the mouse along a path
469+
# Smooth drag (default)
355470
kernel browsers computer drag-mouse <session id> \
356471
--point 100,200 \
357-
--point 150,220 \
358-
--point 200,260 \
359-
--button left \
360-
--delay 0
472+
--point 400,350 \
473+
--point 700,200
474+
475+
# Linear interpolation drag
476+
kernel browsers computer drag-mouse <session id> \
477+
--point 100,200 \
478+
--point 400,350 \
479+
--point 700,200 \
480+
--smooth=false \
481+
--steps-per-segment 10 \
482+
--step-delay-ms 50
361483
```
362484
</CodeGroup>
485+
486+
### Smooth vs linear drag
487+
488+
<Frame caption="Blue = smooth Bezier curves, Red = linear interpolation">
489+
<img src="/images/smooth-drag-demo.gif" />
490+
</Frame>

images/smooth-drag-demo.gif

1.7 MB
Loading

images/smooth-mouse-demo.gif

3.01 MB
Loading

0 commit comments

Comments
 (0)