Skip to content

Commit d988b5b

Browse files
committed
fix(web_fetch): replace hand-written HTML parser with html2text/htmd
- Replace fragile hand-written html_to_text() and html_to_markdown() with html2text (v0.16) and htmd (v0.5) library calls - Fixes UTF-8 multi-byte character panic in html_to_text() - Add test for multi-byte UTF-8 content (Diseño, café, résumé) - Removes ~80 lines of brittle parsing code
1 parent 849ca77 commit d988b5b

3 files changed

Lines changed: 255 additions & 112 deletions

File tree

Cargo.lock

Lines changed: 234 additions & 22 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

core/Cargo.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,10 @@ sha256 = "1.5"
6868
# Shell argument parsing
6969
shell-words = "1.1"
7070

71+
# HTML to text/markdown conversion
72+
html2text = "0.16"
73+
htmd = "0.5"
74+
7175
# Base64 for image encoding
7276
base64 = "0.21"
7377

0 commit comments

Comments
 (0)