Skip to content

Commit 3d7efa3

Browse files
authored
fix: tighten Screen Highlights prompt for atomic extraction (#2294) (#2295)
The shipped prompt was too loose — small models sometimes returned a whole sentence containing the URL, or grouped a label with its value ("Tracking: ABC123" came back as OTHER: Tracking: ABC123). Users tap items to copy them, so each result needs to be a single atomic value. Rewrites the prompt to: - Lead with atomicity ("each entry is a single atomic value") - Inline a worked example per HighlightType — the cheapest reliable way to steer small on-device models - Add an explicit "strip surrounding prose / labels" rule with the Tracking: ABC123 -> OTHER: ABC123 example - Reinforce that multiple URLs on one screen are multiple URL: lines - Tighten OTHER to "atomic identifiers" so it doesn't catch prose No parser, type, cap, backend, UI, or test changes — only the prompt string. Existing 12 parser tests still pass. Closes #2294
1 parent ee9ef9d commit 3d7efa3

1 file changed

Lines changed: 35 additions & 20 deletions

File tree

app/src/main/java/com/enaboapps/switchify/service/llm/ScreenHighlightsTask.kt

Lines changed: 35 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -30,27 +30,42 @@ object ScreenHighlightsTask : AiTask<List<ExtractedItem>> {
3030
private const val ITEMS_CLOSE = "</items>"
3131

3232
override val prompt: String = """
33-
You are extracting actionable information from the screenshot. Think about
34-
what someone using a switch-access input device would want to copy or open
35-
without navigating into the underlying app, then write only the list.
33+
You are extracting key information from the screenshot. The user is going
34+
to tap an item to copy it — so each entry must be a single atomic value,
35+
not a sentence that contains the value.
3636
37-
1. Look across the whole screen — visible text, fields, buttons, anywhere
38-
a useful value might appear.
39-
2. Pull out each piece of useful information you see and label it with one
40-
of these types, in uppercase:
41-
- URL: web links and addresses
42-
- PHONE: phone numbers, in the form shown on screen
43-
- EMAIL: email addresses
44-
- DATE: dates and times
45-
- ADDRESS: street or postal addresses
46-
- OTHER: anything else worth copying — order numbers, tracking codes,
47-
one-time passcodes, reference numbers
48-
3. Use the value exactly as it appears on screen. Do not reformat, expand,
49-
or summarise it.
50-
4. Skip generic UI text, button labels, headings, and anything that would
51-
not be useful to copy.
52-
5. List each item only once. If a value appears twice on the screen,
53-
include it once.
37+
1. Scan the whole screen for individual values worth copying. Each one
38+
becomes its own entry, even when several appear close together.
39+
40+
2. Label each value with one of these types, in uppercase. Each example
41+
shows the on-screen text and the exact line you would write:
42+
- URL — web links and addresses
43+
"see https://example.com/page for details" -> URL: https://example.com/page
44+
- PHONE — phone numbers, in the form shown on screen
45+
"Call 555-123-4567 anytime" -> PHONE: 555-123-4567
46+
- EMAIL — email addresses
47+
"Contact us at support@example.com" -> EMAIL: support@example.com
48+
- DATE — dates and times
49+
"Your delivery is on Tue, 12 Mar at 3pm" -> DATE: Tue, 12 Mar at 3pm
50+
- ADDRESS — street or postal addresses
51+
"Pick up at 1 Infinite Loop, Cupertino" -> ADDRESS: 1 Infinite Loop, Cupertino
52+
- OTHER — atomic identifiers worth copying: order numbers, tracking
53+
codes, one-time passcodes, reference numbers
54+
"Tracking number: ABC-9876" -> OTHER: ABC-9876
55+
56+
3. Each line is one atomic value — never a sentence, label, or prose
57+
that contains the value. Strip surrounding text and punctuation.
58+
"Tracking: ABC123" becomes OTHER: ABC123, not OTHER: Tracking: ABC123.
59+
60+
4. Use the value exactly as it appears on screen. Do not reformat,
61+
expand, or summarise it.
62+
63+
5. List each value only once. If a value appears twice on the screen,
64+
include it once. Multiple distinct URLs on one screen are multiple
65+
URL: lines — pick all of them, not just the first.
66+
67+
6. Skip generic UI text, button labels, headings, and any prose that
68+
contains no atomic value to copy.
5469
5570
Format the answer as exactly this, and nothing else:
5671
$ITEMS_OPEN

0 commit comments

Comments
 (0)