Skip to content

Commit 7ff0c5f

Browse files
tribixbitetribixbite
committed
feat(autocorrect): Tier B layout-aware adjacency + accents + freq fix
Three coordinated changes to the adjacency-aware autocorrect system, plus the fix to the freq-downgrade bug surfaced by Phase A investigation. ## KeyAdjacency — Tier B + accented coverage - Replaces hardcoded `MAX_DISTANCE = q↔m = 7.28` (which was mathematically wrong — true max is `q↔p = 9.0`, same-row opposite ends) with pairwise `computeMaxDistance` that adapts to the active position table. - Adds `setLayout(positions)` / `resetLayout()` so non-QWERTY layouts (AZERTY, QWERTZ, Dvorak, Colemak, user-custom) get adjacency-aware autocorrect using their REAL physical key positions. `@Volatile` snapshot pattern (`val p = positions` at the top of `keyDistance`) is the lightweight thread-safety primitive — `Keyboard2View.onLayout` pushes from the UI thread, autocorrect reads from the prediction thread. - Adds common accented Latin chars (`á à â ä ã å é è ê ë í ì î ï ó ò ô ö õ ø ú ù û ü ñ ç ß ý ÿ`) to the default position table mapped to their unaccented base's position. de/fr/es/it/pt/sv users now get adjacency credit on accent typos (`cafe ↔ café` cost ~0). When the user IS on a layout with dedicated accent keys (German ü/ö/ä, French é/è/à), `setLayout` overrides with the real key positions. ## Keyboard2View.onLayout integration Already exposed `getRealKeyPositions(): Map<Char, PointF>` — never called by anything. Wires it up: after layout finalizes, `KeyAdjacency.setLayout(realPositions)` so the active layout drives autocorrect's distance model. Defensive try/catch so a position-table exception can't kill IME boot. ## WordPredictor selection logic — hybrid score/freq tiebreaker Replaces the previous "alias-key floor frequency = Int.MAX_VALUE / 2" mechanism (which overrode clearly-better non-alias matches) with a four-tier comparator on a new `AutocorrectCandidate(word, score, freq)`: 1. Big score gap (> 0.10) → score wins. Stops `wuestion → within` (lenDiff=2, score 0.69 vs `question` 0.99) and `tge → weve` (alias-keyed but score 0.66 vs `the` 0.96). 2. Alias vs alias → raw score (structural closeness) wins. `hadnr` means `hadnt` not `hasnt` because hadnt is a single adj sub (0.978) vs hasnt's two subs (0.956), regardless of either's dict frequency. 3. Close score (within gap), non-alias → freq wins. Stops `tfe → tfw` (tfw scores 0.96 by 1 adj sub, but `the` at 0.93 freq-beats it 255 vs 162). Same fix lands `questin → question` over `quentin`, and `quuestion → question` over `quotation`. 4. Everything tied → deterministic by raw score (no hash-map iteration-order races). Alias-keys get a +0.15 score bonus so they clear the 0.10 gap on otherwise-tied candidates (`donr → don't` wins by score, not hash-map order). `WordCandidate(word, score: Int)` is kept distinct for the prediction-path use (`predictWords`), which still uses Int unified scores from `calculateUnifiedScore` — the Float score in `AutocorrectCandidate` is the [0,1] match quality from KeyAdjacency, not the same scale at all. ## Freq-downgrade fix in alias-injection `loadPrimaryContractionKeys` (line 1024) and `loadContractionKeysIntoMaps` (line 956) previously did: currentDict[withoutApostrophe] = currentDict[withApostrophe] ?: 5000 This was destructive: the apostrophe form is never in the dict (en_enhanced.json contains no apostrophe entries), so the `?:` ALWAYS fell through to 5000 — silently downgrading `hadnt` from its binary- loaded ≈789K freq to 5000. Fixed to preserve any existing freq: currentDict[withoutApostrophe] = currentDict[withApostrophe] ?: currentDict[withoutApostrophe] ?: 5000 Phase A investigation confirmed safe for beam search: OptimizedVocabulary normalizes freqs to [0,1] then multiplies by `Config.neural_frequency_weight` (user-tunable, default 0.57), so higher input freq → slightly better ranking but no breakage. Beam search reads through `OptimizedVocabulary.WordInfo` not raw dict, so no scale-mixing risk. ## Tests `KeyAdjacencyTest` (+11 cases, total 31): - 7 accent tests covering é/ñ/ç/ü/ß/ý/ä. - 4 layout-injection tests: replaces-default, AZERTY q/a swap, empty-map reverts, case-insensitive normalization. - Updates to 5 pre-existing tests for the corrected MAX = 9.0. Verification: 1242 pure + 194 mock + 1279 instrumented all green. — opus 4.7
1 parent 39b9b52 commit 7ff0c5f

4 files changed

Lines changed: 409 additions & 80 deletions

File tree

‎src/main/kotlin/tribixbite/cleverkeys/Keyboard2View.kt‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1289,6 +1289,20 @@ class Keyboard2View @JvmOverloads constructor(
12891289
)
12901290
systemGestureExclusionRects = listOf(keyboard_area)
12911291
}
1292+
1293+
// Push current layout's key positions to the autocorrect adjacency
1294+
// model so non-QWERTY layouts (AZERTY, QWERTZ, Dvorak, Colemak,
1295+
// user-custom) get adjacency-aware scoring. `getRealKeyPositions`
1296+
// returns an empty map when the layout isn't ready yet — the
1297+
// setter falls back to the QWERTY default in that case.
1298+
try {
1299+
val realPositions = getRealKeyPositions()
1300+
val adjacencyPositions = realPositions.mapValues { (_, pt) -> pt.x to pt.y }
1301+
tribixbite.cleverkeys.autocorrect.KeyAdjacency.setLayout(adjacencyPositions)
1302+
} catch (e: Exception) {
1303+
android.util.Log.w("Keyboard2View",
1304+
"Failed to push layout to KeyAdjacency: ${e.message}")
1305+
}
12921306
}
12931307

12941308
override fun onApplyWindowInsets(wi: WindowInsets?): WindowInsets? {

‎src/main/kotlin/tribixbite/cleverkeys/WordPredictor.kt‎

Lines changed: 123 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -72,26 +72,40 @@ class WordPredictor {
7272
private const val LENGTH_DIFF_ED_BUDGET = 0.5f
7373

7474
/**
75-
* Floor frequency assigned to alias-key candidates (bare-form
76-
* contractions like `dont`, `cant`, `hadnt`) during the dict-scan
77-
* tiebreaker. Sized at `Int.MAX_VALUE / 2` so the alias-key wins
78-
* against ANY non-alias candidate regardless of which freq scale
79-
* is in use (JSON path: ≈100–10k, binary path: ≈5500–1,000,000).
75+
* Score bonus applied to alias-key candidates (bare-form
76+
* contractions like `dont`, `cant`, `hadnt`) in the dict-scan
77+
* tiebreaker. Sized so an alias-key's effective score clears
78+
* [SCORE_TIEBREAK_GAP] above a tied non-alias — that flips ties
79+
* toward the contraction (`donr → don't` instead of `done`)
80+
* without overriding clearly-better non-alias matches.
81+
*/
82+
private const val ALIAS_SCORE_BONUS = 0.15f
83+
84+
/**
85+
* Score-difference threshold for score-vs-frequency tiebreaker.
86+
* When two candidates differ by MORE than this in effective
87+
* score, the higher-scoring one wins regardless of frequency.
88+
* When the gap is within this band, frequency is the tiebreaker
89+
* (the more popular word wins).
8090
*
81-
* Product intent: when a typo is a near-match to a contraction
82-
* base AND clears the score threshold, the contracted form is the
83-
* far more likely intent than a similarly-scored common word —
84-
* `donr → don't` beats `donr → done` because users typing `donr`
85-
* almost always meant `don't`. Tied/near-tied scores are the norm
86-
* for typos: a single adjacent-key substitution lands at score ≈
87-
* 0.97 for many candidate words simultaneously, and the old
88-
* frequency tiebreaker silently picked the wrong winner.
91+
* Why hybrid: same-length multi-substitution candidates have an
92+
* asymmetric scoring advantage over lengthDiff=1 candidates. A
93+
* user typing `quuestion` almost certainly meant `question`
94+
* (one extra letter, ed=1) not `quotation` (three subs, score
95+
* 0.938 vs 0.889 — tantalizingly close by score, semantically
96+
* miles apart). Pure score-primary picks `quotation`; pure
97+
* freq-primary picks `quentin` for `questin`. The 0.10 gap
98+
* threshold separates "clearly better" from "noise" — picks
99+
* `question` correctly via the freq fallback.
89100
*
90-
* Halved from `Int.MAX_VALUE` so two aliases competing in the
91-
* same scan don't overflow into ambiguous wraparound — though
92-
* that case is itself a corner (most typos match only one base).
101+
* Calibrated against:
102+
* - `donr → dont` (alias bonus pushes gap to 0.15) → alias wins
103+
* - `tge → the` (gap 0.149 vs `weve`) → score wins (the)
104+
* - `tfe → the` (gap 0.037 vs `tfw`) → freq wins (the)
105+
* - `quuestion → question` (gap 0.049 vs `quotation`) → freq wins
106+
* - `questin → question` (gap 0.052 vs `quentin`) → freq wins
93107
*/
94-
private const val ALIAS_KEY_FLOOR_FREQUENCY = Int.MAX_VALUE / 2
108+
private const val SCORE_TIEBREAK_GAP = 0.10f
95109
private const val MAX_EDIT_DISTANCE = 2
96110
private const val MAX_RECENT_WORDS = 20 // Keep last 20 words for language detection
97111
private const val PREFIX_INDEX_MAX_LENGTH = 3 // Index prefixes up to 3 chars
@@ -951,9 +965,14 @@ class WordPredictor {
951965
// Skip real English words that are also contraction bases
952966
if (withoutApostrophe in REAL_WORD_CONTRACTION_BASES) continue
953967

954-
// Base form is NOT a real word → create alias and add to dictionary
968+
// Base form is NOT a real word → create alias and add to dictionary.
969+
// Preserve existing freq (same fix as `loadPrimaryContractionKeys`
970+
// — see that function for rationale). The async loader path
971+
// would otherwise silently downgrade bare-form contractions.
955972
aliases[withoutApostrophe] = withApostrophe
956-
targetDict[withoutApostrophe] = targetDict[withApostrophe] ?: 5000
973+
targetDict[withoutApostrophe] = targetDict[withApostrophe]
974+
?: targetDict[withoutApostrophe]
975+
?: 5000
957976

958977
val maxLen = min(PREFIX_INDEX_MAX_LENGTH, withoutApostrophe.length)
959978
for (len in 1..maxLen) {
@@ -1029,9 +1048,24 @@ class WordPredictor {
10291048
if (withoutApostrophe in REAL_WORD_CONTRACTION_BASES) continue
10301049

10311050
// Base form is NOT a real word (e.g., "dont", "im", "thats", "hes")
1032-
// → create autocorrect alias and add to dictionary for predictions
1051+
// → create autocorrect alias and add to dictionary for predictions.
1052+
//
1053+
// Preserve any pre-existing frequency for the bare form (loaded
1054+
// from the binary/JSON dict) rather than overwriting. The
1055+
// previous `?: 5000` form was destructive: a bare form like
1056+
// `hadnt` loaded from binary at ≈ 789K freq would be SILENTLY
1057+
// DOWNGRADED to 5000 (since `currentDict[withApostrophe]` is
1058+
// null — the apostrophe form is never in the dict). The fall-
1059+
// through order is now: apostrophe-form freq if present →
1060+
// existing bare-form freq if present → 5000 anchor. This
1061+
// preserves both the binary-loaded ranking signal and the
1062+
// beam-search ranking (OptimizedVocabulary normalizes to 0-1
1063+
// and multiplies by `Config.neural_frequency_weight`, so
1064+
// higher input freq → slightly better ranking).
10331065
aliases[withoutApostrophe] = withApostrophe
1034-
currentDict[withoutApostrophe] = currentDict[withApostrophe] ?: 5000
1066+
currentDict[withoutApostrophe] = currentDict[withApostrophe]
1067+
?: currentDict[withoutApostrophe]
1068+
?: 5000
10351069

10361070
// Add to prefix index so typing "don" finds "dont" → "don't"
10371071
val maxLen = min(PREFIX_INDEX_MAX_LENGTH, withoutApostrophe.length)
@@ -1791,7 +1825,7 @@ class WordPredictor {
17911825
val maxLengthDiff = (config?.autocorrect_max_length_diff ?: 0).coerceAtLeast(0)
17921826

17931827
// Track top-3 candidates for diagnostic logging on rejection.
1794-
var bestCandidate: WordCandidate? = null
1828+
var bestCandidate: AutocorrectCandidate? = null
17951829
val rejectionLog = mutableListOf<Pair<String, Float>>() // (word, score) for top-N
17961830
val diagnosticsEnabled = config?.swipe_debug_detailed_logging == true
17971831

@@ -1867,28 +1901,52 @@ class WordPredictor {
18671901
}
18681902

18691903
if (score >= charMatchThreshold) {
1870-
// Tiebreaker: higher dictionary frequency wins, but alias-
1871-
// keys (bare-form contractions like `dont`, `cant`, `hadnt`)
1872-
// are floored at `ALIAS_KEY_FLOOR_FREQUENCY` so they always
1873-
// beat non-alias competitors. Among multiple alias-keys
1874-
// (e.g. `couldnr` matches both `couldnt` AND `couldve`),
1875-
// the higher-scoring candidate wins via a score-scaled
1876-
// offset added to the floor. Without the offset, hash-map
1877-
// iteration order silently picks the wrong contraction.
1878-
val effectiveFrequency =
1879-
if (dictWord in contractionAliases) {
1880-
ALIAS_KEY_FLOOR_FREQUENCY + (score * 1000f).toInt()
1881-
} else {
1882-
candidateFrequency
1883-
}
1884-
if (bestCandidate == null || effectiveFrequency > bestCandidate.score) {
1885-
bestCandidate = WordCandidate(dictWord, effectiveFrequency)
1904+
// Two-level tiebreaker:
1905+
// 1. SCORE PRIMARY — higher-scoring candidate wins. This
1906+
// stops a freq-popular but distantly-aligned word from
1907+
// beating a high-quality match: e.g. `wuestion → within`
1908+
// (lenDiff=2 weighted-ed-passes-budget, freq 245)
1909+
// should NOT beat `wuestion → question` (lenDiff=0,
1910+
// near-perfect score 0.986, freq 243).
1911+
// 2. FREQ SECONDARY — when scores are equal (the common
1912+
// "single adjacent-key sub" case where multiple words
1913+
// tie), the more common word wins.
1914+
//
1915+
// Alias-keys get a small `ALIAS_SCORE_BONUS` to flip ties
1916+
// toward the contraction (`donr → don't` not `done`),
1917+
// sized small enough that a clearly-better non-alias still
1918+
// wins (`tge → the` not `we've`).
1919+
val isAlias = dictWord in contractionAliases
1920+
val effectiveScore = if (isAlias) score + ALIAS_SCORE_BONUS else score
1921+
val bothAliases = isAlias && (bestCandidate?.word in contractionAliases)
1922+
val better = when {
1923+
bestCandidate == null -> true
1924+
// Strictly better score by more than the tiebreak gap → win.
1925+
effectiveScore > bestCandidate.effectiveScore + SCORE_TIEBREAK_GAP -> true
1926+
// Strictly worse score by more than the gap → lose.
1927+
effectiveScore < bestCandidate.effectiveScore - SCORE_TIEBREAK_GAP -> false
1928+
// Alias vs alias: structural closeness (raw score) wins, NOT
1929+
// frequency. Sibling contractions (`hadnt` vs `hasnt`) all
1930+
// sit at similar freqs, and typing `hadnr` (d/t-adjacent) means
1931+
// `hadnt` — `hasnt` only matches via TWO substitutions and
1932+
// should lose to the single-typo match regardless of dict
1933+
// popularity.
1934+
bothAliases -> effectiveScore > bestCandidate.effectiveScore
1935+
// Within the gap, normal case → frequency wins (the more
1936+
// popular word for close-quality matches).
1937+
candidateFrequency > bestCandidate.frequency -> true
1938+
candidateFrequency < bestCandidate.frequency -> false
1939+
// Score-close AND freq-tied → deterministic by score.
1940+
else -> effectiveScore > bestCandidate.effectiveScore
1941+
}
1942+
if (better) {
1943+
bestCandidate = AutocorrectCandidate(dictWord, effectiveScore, candidateFrequency)
18861944
}
18871945
}
18881946
}
18891947

18901948
// 5. Apply correction only if confident candidate found.
1891-
if (bestCandidate != null && bestCandidate.score >= frequencyFloor) {
1949+
if (bestCandidate != null && bestCandidate.frequency >= frequencyFloor) {
18921950
// Re-route alias-keyed winners through contractionAliases so the
18931951
// returned form is the apostrophe-bearing contraction. Without
18941952
// this, `donr → dont` (the alias-key) would stop there; the
@@ -1902,7 +1960,9 @@ class WordPredictor {
19021960
} else {
19031961
preserveCapitalization(typedWord, outputWord)
19041962
}
1905-
Log.d(TAG, "AUTO-CORRECT: '$typedWord' → '$corrected' (winner=$winnerWord, freq=${bestCandidate.score})")
1963+
Log.d(TAG, "AUTO-CORRECT: '$typedWord' → '$corrected' " +
1964+
"(winner=$winnerWord score=${"%.3f".format(bestCandidate.effectiveScore)} " +
1965+
"freq=${bestCandidate.frequency})")
19061966
return corrected
19071967
}
19081968

@@ -1913,8 +1973,8 @@ class WordPredictor {
19131973
.joinToString(", ") { "${it.first}=${"%.3f".format(it.second)}" }
19141974
val reason = when {
19151975
bestCandidate == null -> "no candidate above threshold $charMatchThreshold"
1916-
bestCandidate.score < frequencyFloor ->
1917-
"best='${bestCandidate.word}' freq=${bestCandidate.score.toInt()} < floor=$frequencyFloor"
1976+
bestCandidate.frequency < frequencyFloor ->
1977+
"best='${bestCandidate.word}' freq=${bestCandidate.frequency} < floor=$frequencyFloor"
19181978
else -> "?"
19191979
}
19201980
Log.d(TAG, "AUTO-CORRECT-REJECT: '$typedWord' [$reason] top=[$top]")
@@ -1959,10 +2019,30 @@ class WordPredictor {
19592019
}
19602020

19612021
/**
1962-
* Helper class to store word candidates with scores
2022+
* Helper class to store word candidates with scores (used by
2023+
* `predictWords` — score is the unified ranking integer from
2024+
* `calculateUnifiedScore`, NOT a [0,1] match score).
19632025
*/
19642026
private data class WordCandidate(val word: String, val score: Int)
19652027

2028+
/**
2029+
* Helper class to store autocorrect candidates.
2030+
*
2031+
* `effectiveScore` is the adjacency-weighted match quality plus any
2032+
* alias bonus, in [0, ~1.05]. `frequency` is the raw dictionary
2033+
* frequency (scale varies by loader — binary 5K-1M, JSON 100-10K).
2034+
* Selection sorts by effectiveScore desc, then frequency desc.
2035+
*
2036+
* Kept distinct from `WordCandidate` so the prediction path's
2037+
* Int-score contract isn't conflated with the autocorrect path's
2038+
* Float-score-+-Int-freq pair.
2039+
*/
2040+
private data class AutocorrectCandidate(
2041+
val word: String,
2042+
val effectiveScore: Float,
2043+
val frequency: Int
2044+
)
2045+
19662046
/**
19672047
* Result class containing predictions and their scores
19682048
*/

0 commit comments

Comments
 (0)