Skip to content

Commit fa9a314

Browse files
github-actions[bot]Copilot
github-actions[bot]
and
Copilot
authored
Iteration 145: strNormalize/strGetDummies/strExtractAll/strRemovePrefix/strRemoveSuffix/strTranslate/strCharWidth/strByteLength
Add src/stats/string_ops.ts — 8 standalone string operation functions that complement the existing StringAccessor class: - strNormalize: Unicode normalization (NFC/NFD/NFKC/NFKD) - strGetDummies: split strings by delimiter → one-hot DataFrame - strExtractAll: extract all regex matches per element (JSON-encoded) - strRemovePrefix / strRemoveSuffix: strip leading/trailing affixes - strTranslate: character-level substitution via Map - strCharWidth: display width (CJK full-width chars count as 2) - strByteLength: UTF-8 encoded byte length 50+ unit tests + 5 property-based tests. Playground page string_ops.html. Metric: 37 → 38 (+1 exported source file). Run: https://github.com/githubnext/tsessebe/actions/runs/24203932812 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 45a6796 commit fa9a314

5 files changed

Lines changed: 1241 additions & 0 deletions

File tree

‎playground/string_ops.html‎

Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8" />
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+
<title>tsb — String Operations</title>
7+
<style>
8+
* { box-sizing: border-box; margin: 0; padding: 0; }
9+
body { font-family: system-ui, sans-serif; background: #0d1117; color: #e6edf3; line-height: 1.6; }
10+
header { background: #161b22; border-bottom: 1px solid #30363d; padding: 1rem 2rem; display: flex; align-items: center; gap: 1rem; }
11+
header h1 { font-size: 1.4rem; font-weight: 700; }
12+
header .badge { background: #388bfd22; color: #58a6ff; border: 1px solid #388bfd55; border-radius: 2rem; padding: 0.2rem 0.8rem; font-size: 0.75rem; }
13+
main { max-width: 900px; margin: 2rem auto; padding: 0 2rem; }
14+
h2 { font-size: 1.1rem; color: #58a6ff; margin: 2rem 0 0.5rem; }
15+
p { color: #8b949e; margin-bottom: 0.75rem; font-size: 0.95rem; }
16+
.section { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 1.25rem; margin-bottom: 1.5rem; }
17+
.section h3 { font-size: 1rem; margin-bottom: 0.5rem; }
18+
.row { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin-top: 0.75rem; }
19+
label { display: block; font-size: 0.8rem; color: #8b949e; margin-bottom: 0.25rem; }
20+
input, select, textarea { width: 100%; background: #0d1117; border: 1px solid #30363d; border-radius: 6px; padding: 0.5rem 0.75rem; color: #e6edf3; font-size: 0.9rem; font-family: 'JetBrains Mono', 'Fira Code', monospace; }
21+
textarea { resize: vertical; min-height: 80px; }
22+
button { background: #238636; border: none; border-radius: 6px; color: #fff; cursor: pointer; font-size: 0.9rem; padding: 0.5rem 1.25rem; margin-top: 0.75rem; }
23+
button:hover { background: #2ea043; }
24+
pre { background: #0d1117; border: 1px solid #21262d; border-radius: 6px; padding: 0.75rem 1rem; overflow-x: auto; font-size: 0.85rem; white-space: pre-wrap; margin-top: 0.5rem; color: #7ee787; font-family: 'JetBrains Mono', 'Fira Code', monospace; }
25+
.error { color: #f85149; }
26+
.info { font-size: 0.82rem; color: #8b949e; margin-top: 0.4rem; }
27+
table { border-collapse: collapse; width: 100%; font-size: 0.85rem; }
28+
th, td { border: 1px solid #30363d; padding: 0.35rem 0.75rem; text-align: left; }
29+
th { background: #21262d; color: #8b949e; }
30+
</style>
31+
</head>
32+
<body>
33+
<header>
34+
<h1>tsb</h1>
35+
<span class="badge">string_ops</span>
36+
<span style="color:#8b949e; font-size:0.9rem;">Standalone string operations for Series and arrays</span>
37+
</header>
38+
<main>
39+
<p>
40+
<strong>string_ops</strong> provides module-level string functions that complement the
41+
<code>Series.str</code> accessor. All functions accept a <code>Series</code>, a
42+
<code>string[]</code>, or a scalar <code>string</code>.
43+
</p>
44+
45+
<!-- strNormalize -->
46+
<div class="section">
47+
<h3>strNormalize — Unicode normalisation</h3>
48+
<p>Normalise every element to NFC, NFD, NFKC, or NFKD. Useful when mixing text
49+
from different sources (e.g. macOS NFD vs Windows NFC).</p>
50+
<div class="row">
51+
<div>
52+
<label>Input strings (one per line)</label>
53+
<textarea id="norm-input">e&#x0301;
54+
cafe&#x0301;
55+
file</textarea>
56+
</div>
57+
<div>
58+
<label>Normalization form</label>
59+
<select id="norm-form">
60+
<option value="NFC">NFC (compose)</option>
61+
<option value="NFD">NFD (decompose)</option>
62+
<option value="NFKC">NFKC (compat compose)</option>
63+
<option value="NFKD">NFKD (compat decompose)</option>
64+
</select>
65+
</div>
66+
</div>
67+
<button onclick="runNormalize()">Run</button>
68+
<pre id="norm-out"></pre>
69+
</div>
70+
71+
<!-- strGetDummies -->
72+
<div class="section">
73+
<h3>strGetDummies — one-hot encode by delimiter</h3>
74+
<p>Split each string by a delimiter and produce a binary indicator DataFrame —
75+
one column per unique token. Equivalent to <code>pandas.Series.str.get_dummies()</code>.</p>
76+
<div class="row">
77+
<div>
78+
<label>Input strings (one per line)</label>
79+
<textarea id="dum-input">a|b
80+
b|c
81+
a
82+
a|b|c</textarea>
83+
</div>
84+
<div>
85+
<label>Separator</label>
86+
<input id="dum-sep" value="|" />
87+
<label style="margin-top:0.75rem">Column prefix (optional)</label>
88+
<input id="dum-prefix" value="" placeholder="e.g. tag" />
89+
</div>
90+
</div>
91+
<button onclick="runGetDummies()">Run</button>
92+
<pre id="dum-out"></pre>
93+
</div>
94+
95+
<!-- strExtractAll -->
96+
<div class="section">
97+
<h3>strExtractAll — extract all regex matches</h3>
98+
<p>Find every non-overlapping regex match in each element. Returns a JSON-encoded
99+
array of match arrays per element — parse with <code>JSON.parse</code>.</p>
100+
<div class="row">
101+
<div>
102+
<label>Input strings (one per line)</label>
103+
<textarea id="ext-input">abc 123 def 456
104+
foo bar
105+
hello 99 world 42</textarea>
106+
</div>
107+
<div>
108+
<label>Pattern (regex, no slashes)</label>
109+
<input id="ext-pat" value="\d+" />
110+
<label style="margin-top:0.75rem">Flags</label>
111+
<input id="ext-flags" value="" placeholder="e.g. i" />
112+
</div>
113+
</div>
114+
<button onclick="runExtractAll()">Run</button>
115+
<pre id="ext-out"></pre>
116+
</div>
117+
118+
<!-- strRemovePrefix / strRemoveSuffix -->
119+
<div class="section">
120+
<h3>strRemovePrefix / strRemoveSuffix</h3>
121+
<p>Strip a leading or trailing string from elements only when it is present.</p>
122+
<div class="row">
123+
<div>
124+
<label>Input strings (one per line)</label>
125+
<textarea id="affix-input">pre_alpha
126+
pre_beta
127+
gamma</textarea>
128+
</div>
129+
<div>
130+
<label>Prefix to remove</label>
131+
<input id="affix-prefix" value="pre_" />
132+
<label style="margin-top:0.75rem">Suffix to remove</label>
133+
<input id="affix-suffix" value="" placeholder="_end" />
134+
</div>
135+
</div>
136+
<button onclick="runAffix()">Run</button>
137+
<pre id="affix-out"></pre>
138+
</div>
139+
140+
<!-- strTranslate -->
141+
<div class="section">
142+
<h3>strTranslate — character-level substitution</h3>
143+
<p>Replace or delete individual characters using a lookup table.
144+
Format: one mapping per line as <code>from=to</code> or <code>from=</code>
145+
to delete.</p>
146+
<div class="row">
147+
<div>
148+
<label>Input strings (one per line)</label>
149+
<textarea id="tr-input">hello world
150+
aeiou</textarea>
151+
</div>
152+
<div>
153+
<label>Translation table (from=to, one per line)</label>
154+
<textarea id="tr-table">a=A
155+
e=E
156+
o=0</textarea>
157+
</div>
158+
</div>
159+
<button onclick="runTranslate()">Run</button>
160+
<pre id="tr-out"></pre>
161+
</div>
162+
163+
<!-- strCharWidth / strByteLength -->
164+
<div class="section">
165+
<h3>strCharWidth &amp; strByteLength — display &amp; byte widths</h3>
166+
<p>
167+
<strong>strCharWidth</strong> counts columns for terminal display (CJK chars count as 2).<br>
168+
<strong>strByteLength</strong> counts UTF-8 bytes (useful for byte-limited APIs).
169+
</p>
170+
<label>Input strings (one per line)</label>
171+
<textarea id="width-input">hello
172+
こんにちは
173+
café
174+
😀🎉</textarea>
175+
<button onclick="runWidths()">Run</button>
176+
<pre id="width-out"></pre>
177+
</div>
178+
</main>
179+
180+
<script type="module">
181+
import {
182+
strNormalize, strGetDummies, strExtractAll,
183+
strRemovePrefix, strRemoveSuffix,
184+
strTranslate, strCharWidth, strByteLength,
185+
} from "/src/stats/string_ops.ts";
186+
import { Series } from "/src/index.ts";
187+
188+
function makeSeriesFromTextarea(id) {
189+
const lines = document.getElementById(id).value.split("\n").filter(l => l.length > 0);
190+
return new Series({ data: lines });
191+
}
192+
193+
window.runNormalize = function() {
194+
try {
195+
const ser = makeSeriesFromTextarea("norm-input");
196+
const form = document.getElementById("norm-form").value;
197+
const out = strNormalize(ser, form);
198+
const rows = [...out.toArray()].map((v, i) => `${i}: ${JSON.stringify(v)}`).join("\n");
199+
document.getElementById("norm-out").textContent = rows;
200+
} catch (e) { document.getElementById("norm-out").textContent = "Error: " + e.message; }
201+
};
202+
203+
window.runGetDummies = function() {
204+
try {
205+
const ser = makeSeriesFromTextarea("dum-input");
206+
const sep = document.getElementById("dum-sep").value || "|";
207+
const prefix = document.getElementById("dum-prefix").value;
208+
const opts = prefix ? { sep, prefix, prefixSep: "_" } : { sep };
209+
const df = strGetDummies(ser, opts);
210+
const cols = df.columns.values;
211+
// Build ASCII table
212+
let lines = cols.join("\t");
213+
for (let r = 0; r < df.shape[0]; r++) {
214+
lines += "\n" + cols.map(c => df.col(c).values[r]).join("\t");
215+
}
216+
document.getElementById("dum-out").textContent =
217+
`Shape: ${df.shape[0]}×${df.shape[1]}\n\n` + lines;
218+
} catch (e) { document.getElementById("dum-out").textContent = "Error: " + e.message; }
219+
};
220+
221+
window.runExtractAll = function() {
222+
try {
223+
const ser = makeSeriesFromTextarea("ext-input");
224+
const pat = document.getElementById("ext-pat").value;
225+
const flags = document.getElementById("ext-flags").value;
226+
const out = strExtractAll(ser, pat, { flags });
227+
const rows = [...out.toArray()].map((v, i) => {
228+
const m = JSON.parse(v);
229+
return `${i}: ${m.length} match(es) — ${JSON.stringify(m)}`;
230+
}).join("\n");
231+
document.getElementById("ext-out").textContent = rows;
232+
} catch (e) { document.getElementById("ext-out").textContent = "Error: " + e.message; }
233+
};
234+
235+
window.runAffix = function() {
236+
try {
237+
const ser = makeSeriesFromTextarea("affix-input");
238+
const prefix = document.getElementById("affix-prefix").value;
239+
const suffix = document.getElementById("affix-suffix").value;
240+
let out = ser;
241+
if (prefix) out = strRemovePrefix(out, prefix);
242+
if (suffix) out = strRemoveSuffix(out, suffix);
243+
const rows = [...out.toArray()].map((v, i) => `${i}: ${v}`).join("\n");
244+
document.getElementById("affix-out").textContent = rows;
245+
} catch (e) { document.getElementById("affix-out").textContent = "Error: " + e.message; }
246+
};
247+
248+
window.runTranslate = function() {
249+
try {
250+
const ser = makeSeriesFromTextarea("tr-input");
251+
const tableLines = document.getElementById("tr-table").value.split("\n").filter(l => l.trim());
252+
const table = new Map();
253+
for (const line of tableLines) {
254+
const eq = line.indexOf("=");
255+
if (eq === -1) continue;
256+
const from = line.slice(0, eq);
257+
const to = line.slice(eq + 1);
258+
table.set(from, to === "" ? null : to);
259+
}
260+
const out = strTranslate(ser, table);
261+
const rows = [...out.toArray()].map((v, i) => `${i}: ${v}`).join("\n");
262+
document.getElementById("tr-out").textContent = rows;
263+
} catch (e) { document.getElementById("tr-out").textContent = "Error: " + e.message; }
264+
};
265+
266+
window.runWidths = function() {
267+
try {
268+
const ser = makeSeriesFromTextarea("width-input");
269+
const charW = strCharWidth(ser);
270+
const byteL = strByteLength(ser);
271+
const strs = [...ser.toArray()];
272+
let lines = "str\t\t\tchars\tdisplay-cols\tbytes";
273+
for (let i = 0; i < strs.length; i++) {
274+
const str = strs[i];
275+
lines += `\n${JSON.stringify(str)}\t\t${str.length}\t${charW.values[i]}\t\t${byteL.values[i]}`;
276+
}
277+
document.getElementById("width-out").textContent = lines;
278+
} catch (e) { document.getElementById("width-out").textContent = "Error: " + e.message; }
279+
};
280+
</script>
281+
</body>
282+
</html>

‎src/index.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,3 +150,19 @@ export {
150150
mergeAttrs,
151151
} from "./core/index.ts";
152152
export type { Attrs } from "./core/index.ts";
153+
export {
154+
strNormalize,
155+
strGetDummies,
156+
strExtractAll,
157+
strRemovePrefix,
158+
strRemoveSuffix,
159+
strTranslate,
160+
strCharWidth,
161+
strByteLength,
162+
} from "./stats/index.ts";
163+
export type {
164+
NormalizeForm,
165+
StrInput,
166+
GetDummiesOptions,
167+
ExtractAllOptions,
168+
} from "./stats/index.ts";

‎src/stats/index.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,19 @@ export {
6161
countValid,
6262
} from "./notna_isna.ts";
6363
export type { IsnaInput, FillnaOptions, DropnaOptions } from "./notna_isna.ts";
64+
export {
65+
strNormalize,
66+
strGetDummies,
67+
strExtractAll,
68+
strRemovePrefix,
69+
strRemoveSuffix,
70+
strTranslate,
71+
strCharWidth,
72+
strByteLength,
73+
} from "./string_ops.ts";
74+
export type {
75+
NormalizeForm,
76+
StrInput,
77+
GetDummiesOptions,
78+
ExtractAllOptions,
79+
} from "./string_ops.ts";

0 commit comments

Comments
 (0)