Skip to content

Commit d0d3d4e

Browse files
committed
feat(programmer): Rbx-only world building, Hub sub-tabs, vision self-probe
Programmer surface: drop the low-level coreai_world_* build APIs (spawn/change/ set_color/destroy/spawn_batch) from persistent mods and the one-off executor via RegisterWorldEditBuildBindings=false; world building now goes through the Rbx surface (Instance.new, game/workspace, Vector3/CFrame/Color3), while the WorldEdit capability still gates Rbx and the read-only coreai_world_find/pos/exists queries remain. Reworked Programmer.txt / C# fallback / LuaTool / MCP tool descriptions, the Lua Modding skill and the gallery example Lua to match, and added read_skill( 'Full Lua') for the rarely-needed unity_* reflection backup. Hub: reusable HubSubTabView/HubSubTabPage; Settings/Token Budget/Statistics grouped under one AI Settings tab. Vision Detect self-probe (VisionSelfProbe) auto-sets the vision gate from a synthetic-image round-trip. Lua syntax highlighting in the inline mod editor. Endpoint editor session-key field relabelled 'Session API key' to avoid colliding with the HTTP quick-config 'API key'. Tests: pin RegisterWorldEditBuildBindings=false (build APIs gone, Rbx + queries survive); re-point endpoint-editor UI tests to the per-row Edit/Remove redesign; Full Lua skill + Rbx-migrated prompt/skill coverage. 2209/2209 EditMode green.
1 parent e249c5c commit d0d3d4e

42 files changed

Lines changed: 1546 additions & 217 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

Assets/CoreAI.Demos/FullAccess/Scripts/FullAccessHubDemoController.cs

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -57,24 +57,23 @@ private void Awake()
5757
LuaPlatformHubPage.DefaultPageId,
5858
() => new LuaPlatformHubPage(ResolveDriver),
5959
20);
60-
Registry.Register(
61-
TokenBudgetHubPage.DefaultPageId,
62-
() => new TokenBudgetHubPage(),
63-
90);
6460

6561
// WHY: no Chat tab here — this scene keeps its dedicated CoreAiChatUI panel. The Hub is the
66-
// tools companion: Settings (the UITK backend config that replaces CoreAiBackendPanel) and
67-
// Statistics, both fed live values from the scene scope when available (null-tolerant).
62+
// tools companion. Settings, Token Budget and Statistics are grouped under one "AI Settings" tab
63+
// as sub-tabs (HubSubTabPage) so the top tab bar stays short; each is fed live scope values when
64+
// available (null-tolerant).
6865
ICoreAISettings settings = ResolveFromCore<ICoreAISettings>();
6966
InMemoryAiOrchestrationMetrics metrics = ResolveFromCore<InMemoryAiOrchestrationMetrics>();
7067
Registry.Register(
71-
HubSettingsPage.DefaultPageId,
72-
() => new HubSettingsPage(settings, chatConfig),
68+
"coreai.hub.aisettings",
69+
() => new HubSubTabPage(
70+
"coreai.hub.aisettings",
71+
"AI Settings",
72+
100,
73+
new HubSettingsPage(settings, chatConfig),
74+
new TokenBudgetHubPage(),
75+
new HubStatisticsPage(metrics, settings)),
7376
100);
74-
Registry.Register(
75-
HubStatisticsPage.DefaultPageId,
76-
() => new HubStatisticsPage(metrics, settings),
77-
200);
7877

7978
CoreAiHubWindow window = GetComponent<CoreAiHubWindow>();
8079
if (window != null)

Assets/CoreAI/CHANGELOG.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,27 @@
1010
- Hub **AI Settings → Vision** override (Auto / On / Off): forces the vision/camera gate on a multimodal
1111
model whose name the auto-heuristic does not recognise (e.g. a local `qwen3.5` vision build), so the
1212
camera tool and image sends become usable without renaming the model. (`CoreAISettingsAsset.SetVisionSupport`)
13+
- Hub **AI Settings → Vision → Detect**: a self-probe sends the model a synthetic image and reads back
14+
the answer to decide whether it can actually see, then sets the Vision gate accordingly — so Auto
15+
works even for a vision model the name heuristic misses. (`VisionSelfProbe`)
16+
- Hub **sub-tabs**: a reusable `HubSubTabView` / `HubSubTabPage` groups several pages under one top tab.
17+
**Settings / Token Budget / Statistics** are now sub-tabs of a single **AI Settings** tab, keeping the
18+
top tab bar short; new sub-tab groupings are one registration.
19+
- Hub **Lua mod editor** now renders **Lua syntax highlighting** in the inline editor (keywords,
20+
strings, numbers, comments) while keeping the field editable. (`LuaSyntaxHighlighter`)
21+
- Programmer skill **`read_skill('Full Lua')`**: the full `unity_*` reflection surface moved behind an
22+
on-demand skill, so it stays available as a rarely-needed backup without bloating the base prompt.
1323

1424
### Changed
1525

26+
- **The Programmer builds the world Roblox-style only.** The low-level `coreai_world_*` build APIs
27+
(`spawn` / `change` / `set_color` / `destroy` / `spawn_batch`) are no longer bound for persistent mods
28+
or the one-off `execute_lua` executor; world building goes through the Rbx surface
29+
(`Instance.new('Part')`, `game` / `workspace`, `Vector3` / `CFrame` / `Color3`). The read-only queries
30+
`coreai_world_find` / `coreai_world_pos` / `coreai_world_exists` remain, and the `WorldEdit` capability
31+
still gates the Rbx surface. Prompts, tool descriptions, the Lua Modding skill and the gallery example
32+
Lua were updated to match. (`RegisterWorldEditBuildBindings`)
33+
1634
- Hub **API-profiles endpoint editor** redesigned: field visibility is applied on first render (HTTP
1735
endpoints no longer show LLMUnity-only fields), an **Advanced** foldout hides secondary fields
1836
(Endpoint ID, context window, ports, slots, secret reference, LLMUnity options), text fields carry

Assets/CoreAI/Docs/LLM_TOOLS.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,12 @@ resolved value is `CoreAISettingsAsset.IsVisionEnabled`, surfaced as `CoreAi.IsV
107107
`CoreAiChatService.IsVisionEnabled()`. **Both** the host send path and tool registration check it; when it
108108
is `false`, no image is attached and `capture_camera` is not registered.
109109

110+
The Hub **AI Settings** page exposes this as a **Vision** dropdown (Auto / On / Off, backed by
111+
`CoreAISettingsAsset.SetVisionSupport`). Use **On** to force the gate open for a multimodal model whose
112+
name the `Auto` heuristic does not recognise (e.g. a local `qwen3.5` vision build), so the camera tool and
113+
image sends become usable without renaming the model; use **Off** to hard-disable vision on a model the
114+
heuristic would otherwise treat as vision-capable.
115+
110116
### Host send path (camera → model)
111117

112118
The provider-safe one-shot path: capture a camera and send it as a single **user** message

Assets/CoreAI/Runtime/Core/Features/AgentPrompts/BuiltInAgentSystemPromptTexts.cs

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,10 @@ internal static class BuiltInAgentSystemPromptTexts
3939
internal const string Programmer =
4040
"You are the Programmer agent for the CoreAI Lua sandbox (Lua-CSharp runtime). " +
4141
"Use the execute_lua tool to run Lua code; use manage_mods (list/get_source/load/reload/unload) for persistent mods with hooks. " +
42-
"Answer plain questions, greetings, and one-off replies directly - do NOT call read_skill or any tool for those. Only when the task actually needs the deeper API (a game loop, timers, cross-mod communication, input handling), call read_skill('Lua Modding') once first - it returns the full API reference with worked examples; the list below is only the survival minimum. For Roblox-style scripts (Instance.new, game/workspace, Vector3/CFrame/Color3, Part properties), call read_skill('Rbx API') instead. " +
43-
"Typical globals when the game wires them: report(msg), logic_list(), logic_define(name, fn), logic_reset(name) for game-rule slots; " +
44-
"coreai_world_spawn({prefab,name,x,y,z,rx,ry,rz,scale,scaleX,scaleY,scaleZ,parent}), coreai_world_change(name,{x,y,z,rx,ry,rz,scale,scaleX,scaleY,scaleZ,parent}), coreai_world_set_color, and coreai_world_destroy for world changes. Call logic_list() when unsure which rule slots exist. " +
45-
"Full Lua Mode skill: when Full is enabled, use unity_* APIs only after a one-shot diagnostic execute_lua call, read its Success/Output/Error, then load or reload a persistent mod if needed. " +
46-
"For scene objects prefer unity_list_objects(max), unity_find_all(pattern,max), unity_find_by_tag(tag,max), unity_find_by_component(type,max), unity_describe_object(id), unity_get_transform(id), unity_set_position(id,x,y,z), unity_set_rotation_euler(id,x,y,z), unity_set_scale(id,x,y,z), unity_parent(child,parent,worldPositionStays), unity_get_children(id), unity_list_components(id), unity_get_member(id,component,member), unity_set_member(id,component,member,value), and unity_call(id,component,method,args). " +
47-
"WorldEdit APIs do not require Full mode; for visible spawns, call coreai_world_list_prefabs first, then coreai_world_spawn({ prefab='key', name='objectName', x=0, y=0, z=0 }) or coreai_world_spawn_batch with a real prefab key; primitives 'cube'/'sphere'/'cylinder'/'capsule'/'plane'/'empty' also work as prefab keys; report() alone is not a spawn. " +
42+
"Answer plain questions, greetings, and one-off replies directly - do NOT call read_skill or any tool for those. Only when the task actually needs the deeper API (a game loop, timers, cross-mod communication, input handling), call read_skill('Lua Modding') once first - it returns the full API reference with worked examples; the list below is only the survival minimum. For building anything visible in the world - parts, models, whole scenes - call read_skill('Rbx API') and build Roblox-style: Instance.new('Part'), game/workspace, Vector3/CFrame/Color3, Part properties. The Rbx surface is THE way to create and edit world objects. " +
43+
"Typical globals when the game wires them: report(msg), logic_list(), logic_define(name, fn), logic_reset(name) for game-rule slots. Call logic_list() when unsure which rule slots exist. " +
44+
"To inspect the existing scene use the read-only queries coreai_world_find(pattern), coreai_world_pos(name), and coreai_world_exists(name) - good for locating objects including the camera. A report() line alone is not a scene change. " +
45+
"Full reflection (unity_* scene APIs) is a rarely-needed backup - when a task truly needs it, call read_skill('Full Lua') first. " +
4846
"Inside mods you also have: hooks_every(seconds, fn) repeating timers (min 0.05s) and hooks_on('tick', fn) per-frame alias for game loops; store_set(key, value)/store_get(key) per-mod persistent strings (survive restarts); events_emit(name, payload) to other mods and hooks_on(name, function(evt, payload) ... end) to receive; mods_export(name, valueOrFn), mods_get(otherModId, name), mods_call(otherModId, fnName, ...), mods_list_exports(otherModId) for cross-mod variables and function calls (values copy by value, primitives and plain tables only); input_key('a')/input_key_down/input_key_up, input_mouse_button(0), input_mouse_x()/input_mouse_y(), input_axis('Horizontal') to read player input (poll held keys from timers - whole game loops like a falling-blocks game are possible in one mod). " +
4947
"Do not hard-code visual recipes: inspect the scene/components first, then use the smallest real API that matches the host. " +
5048
"Do not invent Lua globals; if a helper is not listed by the task, tool contract, or logic_list/world docs, do not call it. " +
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
namespace CoreAI.Ai
2+
{
3+
/// <summary>
4+
/// Built-in "Full Lua" skill: the model-facing reference for the Full-tier reflection surface
5+
/// (<c>unity_*</c> scene APIs), loaded on demand via <c>read_skill</c>. Mirrors
6+
/// <see cref="BuiltInLuaModdingSkillText"/>: the Programmer prompt only names the skill as a
7+
/// rarely-needed backup; this text carries the whole reference. An optional
8+
/// <c>Resources/AgentSkills/FullLua</c> TextAsset overrides <see cref="Instructions"/>.
9+
/// </summary>
10+
public static class BuiltInFullLuaSkillText
11+
{
12+
/// <summary>Catalog name (what the model passes to <c>read_skill</c>).</summary>
13+
public const string SkillName = "Full Lua";
14+
15+
/// <summary>One-line catalog description shown before the model decides to read the skill.</summary>
16+
public const string SkillDescription =
17+
"Full-tier reflection over arbitrary scene objects and components: unity_list_objects/" +
18+
"unity_find_*/unity_describe_object, transform setters, unity_get_member/unity_set_member/" +
19+
"unity_call. A rarely-needed backup for tasks the Rbx surface and Lua mechanics cannot " +
20+
"cover; requires Full mode to be enabled. Read only when a task truly needs raw reflection.";
21+
22+
/// <summary>Full instructions returned by <c>read_skill("Full Lua")</c>.</summary>
23+
public const string Instructions = @"# CoreAI Full Lua Reference (Full-tier reflection)
24+
25+
Full mode exposes raw reflection over every GameObject and component in the scene. It is a
26+
BACKUP surface: prefer the Rbx API (read_skill('Rbx API')) for building and the standard mod
27+
mechanics (hooks/timers/store/input) for behaviour. Use Full only when a task truly needs to
28+
inspect or mutate arbitrary existing scene objects. These APIs are available only when the host
29+
enabled Full mode; otherwise every unity_* call is absent.
30+
31+
## Workflow
32+
33+
First run a small diagnostic execute_lua script: inspect the scene, return a compact string or
34+
JSON through return/report, then read Success/Output/Error BEFORE changing anything or loading a
35+
persistent mod. Do not hard-code one recipe for visual requests: inspect objects/components
36+
first, then use the smallest real API that matches the scene.
37+
38+
## Discovery
39+
40+
- `unity_list_objects(max)` -> root objects with ids.
41+
- `unity_find_all(pattern, max)` -> objects whose name matches the pattern.
42+
- `unity_find_by_tag(tag, max)`; `unity_find_by_component(type, max)`.
43+
- `unity_describe_object(id)` -> name, tag, active state, components.
44+
- `unity_get_children(id)`; `unity_list_components(id)`.
45+
46+
## Transforms
47+
48+
- `unity_get_transform(id)` -> position/rotation/scale.
49+
- `unity_set_position(id, x, y, z)`; `unity_set_rotation_euler(id, x, y, z)`;
50+
`unity_set_scale(id, x, y, z)`.
51+
- `unity_parent(childId, parentIdOr0, worldPositionStays)` — parentId 0 unparents.
52+
53+
## Members & methods
54+
55+
- `unity_get_member(id, component, member)` / `unity_set_member(id, component, member, value)`.
56+
- `unity_call(id, component, method, args)` — invoke a component method with an args table.
57+
58+
## Example diagnostic
59+
60+
```lua
61+
local objs = unity_find_all('Enemy', 10)
62+
local names = {}
63+
for i, o in ipairs(objs) do names[i] = o.name .. ':' .. o.id end
64+
report(table.concat(names, ','))
65+
```
66+
Read the Output, pick the exact ids, and only then apply the smallest mutation
67+
(unity_set_position / unity_set_member) that fulfils the request.";
68+
}
69+
}

Assets/CoreAI/Runtime/Core/Features/AgentPrompts/BuiltInFullLuaSkillText.cs.meta

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Assets/CoreAI/Runtime/Core/Features/AgentPrompts/BuiltInLuaModdingSkillText.cs

Lines changed: 33 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -48,20 +48,17 @@ and type annotations/casts all work. Plain Lua 5.2 is unaffected.
4848
4949
## 2. World API (WorldEdit tier - no Full mode needed)
5050
51-
- `coreai_world_list_prefabs()` -> table of prefab keys. Call before the first spawn.
52-
- `coreai_world_spawn({prefab='key', name='obj1', x=0,y=0,z=0, rx=0,ry=0,rz=0,
53-
scale=1 or scaleX/scaleY/scaleZ, parent='optionalParentName'})`
54-
Primitives work as prefab keys: 'cube','sphere','cylinder','capsule','plane','empty'.
55-
- `coreai_world_change(name, {x=,y=,z=, rx=,ry=,rz=, scale=, parent=})` - move/rotate/rescale.
56-
- `coreai_world_set_color(name, '#RRGGBB')`
57-
- `coreai_world_destroy(name)`; `coreai_world_exists(name)` -> bool
58-
- `coreai_world_spawn_batch(list)` and `coreai_world_grid` for many objects at once.
59-
- Transactions: `coreai_world_begin_transaction()` / `commit` / `rollback` to group edits.
60-
- Names are the contract: pick unique names at spawn and drive everything by name afterwards.
51+
World BUILDING (creating, moving, recoloring, destroying objects) is done Roblox-style:
52+
`Instance.new('Part')`, `game`/`workspace`, `Vector3`/`CFrame`/`Color3`, and Part properties
53+
(Position, Size, Color, Anchored, Parent = workspace). Call `read_skill('Rbx API')` for the
54+
full reference before building anything visible - there is no other world-building surface.
6155
6256
## 3. Scene queries (read-only, no Full mode needed)
6357
64-
- `coreai_world_count_objects()`, `coreai_world_find(name)`, `coreai_world_get_transform(name)`
58+
- `coreai_world_find(pattern)` -> list of scene object names containing the pattern
59+
(case-insensitive substring match, at most 100 results).
60+
- `coreai_world_pos(name)` -> `{x=,y=,z=}` table, or nil when the object does not exist.
61+
- `coreai_world_exists(name)` -> bool.
6562
(availability depends on host wiring - if a query global is nil, fall back to unity_* under Full).
6663
6764
## 4. Full tier (only when the host enabled Full Lua)
@@ -75,7 +72,7 @@ and type annotations/casts all work. Plain Lua 5.2 is unaffected.
7572
`unity_parent(childId, parentIdOr0, worldPositionStays)`
7673
- `unity_get_member(id,component,member)`, `unity_set_member(id,component,member,value)`,
7774
`unity_call(id,component,method,argsTable)`
78-
Prefer coreai_world_* for plain transforms even in Full mode - it is cheaper and logged.
75+
Prefer the Rbx surface (Part.Position / Part.CFrame) for plain transforms even in Full mode.
7976
8077
## 5. Timers & game loops (inside mods)
8178
@@ -135,42 +132,45 @@ fired an exact number of times - count what actually happened.
135132
136133
## 11. Complete mod example - miniature falling-block game
137134
138-
Shows: spawn, color, per-load unique names, input with edge detection, gravity accumulator,
139-
store persistence, an event another mod (or the host) can send.
135+
Shows: Rbx-style build (Instance.new / workspace), reload-safe root, input with edge
136+
detection, gravity accumulator, store persistence, an event another mod (or the host) can send.
140137
141138
```lua
142-
-- Per-load generation suffix: Unity destroys deferred (end of frame), so a reload that
143-
-- destroys the old objects and spawns identical names in the same frame would collide.
144-
local gen = (tonumber(store_get('gen')) or 0) + 1
145-
store_set('gen', tostring(gen))
146-
local ROOT = 'MiniGameRoot_g' .. gen
147-
local prev = 'MiniGameRoot_g' .. (gen - 1)
148-
if coreai_world_exists(prev) then coreai_world_destroy(prev) end
149-
coreai_world_spawn({ prefab = 'empty', name = ROOT, x = 0, y = 0, z = 0 })
139+
-- Reload-safe root: drop the previous generation before building this one.
140+
local prev = workspace:FindFirstChild('MiniGameRoot')
141+
if prev then prev:Destroy() end
142+
local root = Instance.new('Model')
143+
root.Name = 'MiniGameRoot'
144+
root.Parent = workspace
150145
151146
local score = tonumber(store_get('score')) or 0
152-
local px, py = 0, 10
153-
coreai_world_spawn({ prefab = 'cube', name = ROOT .. '_p', parent = ROOT, x = px, y = py, z = 0 })
154-
coreai_world_set_color(ROOT .. '_p', '#00CCFF')
147+
local px, py = 0, 40 -- studs (1 stud = 0.28 m)
148+
local player = Instance.new('Part')
149+
player.Name = 'MiniGamePlayer'
150+
player.Size = Vector3.new(4, 4, 4)
151+
player.Position = Vector3.new(px, py, 0)
152+
player.Color = Color3.fromHex('#00CCFF')
153+
player.Anchored = true
154+
player.Parent = root
155155
156156
local was_w = false
157157
hooks_every(0.05, function() -- input at 20 Hz
158-
if input_key('a') and px > -4 then px = px - 1 end
159-
if input_key('d') and px < 4 then px = px + 1 end
158+
if input_key('a') and px > -16 then px = px - 4 end
159+
if input_key('d') and px < 16 then px = px + 4 end
160160
local w = input_key('w') -- derived edge, never input_key_down here
161161
if w and not was_w then score = score + 1 end
162162
was_w = w
163-
coreai_world_change(ROOT .. '_p', { x = px, y = py, z = 0 })
163+
player.Position = Vector3.new(px, py, 0)
164164
end)
165165
166166
local acc = 0
167167
hooks_every(0.1, function() -- gravity with accumulator
168168
acc = acc + (input_key('s') and 5 or 1)
169169
if acc < 5 then return end
170170
acc = 0
171-
py = py - 1
171+
py = py - 4
172172
if py <= 0 then
173-
py = 10
173+
py = 40
174174
score = score + 10
175175
store_set('score', tostring(score))
176176
report('landed, score=' .. score)
@@ -196,7 +196,8 @@ fired an exact number of times - count what actually happened.
196196
- hooks_every interval below 0.05 s is clamped - do not rely on faster timers.
197197
- report() shows nothing: report logging is disabled for the mod by default - it is a host
198198
switch, not an error; state still changes.
199-
- A spawn you cannot see: wrong prefab key - call coreai_world_list_prefabs() first;
200-
primitives 'cube'/'sphere'/'cylinder'/'capsule'/'plane'/'empty' always work.";
199+
- A Part you cannot see: set .Parent (usually workspace) after configuring it and set
200+
Anchored = true; remember coordinates are in STUDS (1 stud = 0.28 m), so meter-sized
201+
numbers build a tiny scene.";
201202
}
202203
}

0 commit comments

Comments
 (0)