Dripper VLM mislabels sidebar and spam nodes as main content on CSDN pages

[Jack47]

Problem

When processing CSDN (blog.csdn.net) pages, the Dripper VLM model occasionally assigns _item_id annotations to noise DOM nodes (sidebar recommendations, spam links), classifying them as "main" content. This causes main_html to include large amounts of non-article content.

We tested on a sample of 240 CSDN pages from production. 3 pages (1.2%) have real content defects where noise leaks into the extracted main_html.

Reproduction Cases

Case 1: Sidebar recommendation leak (2/240 pages)

URL: https://blog.csdn.net/goldfishsky/article/details/149716564/

Steps:

1. Run MinerUHTML.process(html) on this page's HTML
2. Inspect _item_id annotations in the output main_html
3. Observe: _item_id 3, 12–37 correctly annotate the article body (<div class="content_views">)
4. Bug: _item_id 38–81 annotate the CSDN recommendation sidebar (<div id="recommend">)

Impact: The article body is 2,869 chars (4.7% of main_html). The sidebar is 58,106 chars (95.3%). The extracted content is 95% noise.

HTML at the boundary:

<!-- Last article node (correct) -->
<p _item_id="37">如果输出 <code>2147483647</code>，则表示无限强度策略已生效。</p>
</div></div></div>

<!-- Sidebar starts here (should NOT have _item_id) -->
<div id="recommend">
    <div class="recommend_list">
        <dl class="container-fluid" data-url="https://download.csdn.net/...">
            <!-- _item_id="38" assigned to first recommend item -->

A second page exhibits the same pattern: https://blog.csdn.net/Angellyouran/article/details/147597019/

Case 2: GitCode spam link leak (1/240 pages)

URL: https://blog.csdn.net/gitblog_00820/article/details/152394486

Steps:

1. Run MinerUHTML.process(html) on this page's HTML
2. Inspect the spam <a> tag with _item_id=13

Observation: The model assigned _item_id=13 to a CSDN-injected GitCode spam link:

<a _item_id="13" class="has-card" href="https://gitcode.com/GitHub_Trending/ra/raylib/...">
  <span class="link-title">【免费下载链接】raylib</span>
</a>

This spam text (【免费下载链接】) appears in the final extracted content.

Context: 123/240 CSDN pages in our sample contain GitCode spam links. The model correctly excludes them in 120/123 cases (97.6% accuracy). This is one of 3 leak cases.

Case 3: Cross-domain sidebar noise

The sidebar issue is not CSDN-specific. In our 2,000-page sample across multiple domains:

Domain	Pages with <aside> / sidebar noise in main_html	Total pages
gitmemories.com	5	67
go.dev	5	6
csdn.net	2	240
arcgis.com	1	2
Several others	1 each	—

Root Cause Analysis

simplify_html.py strips <nav>, <script>, <style>, etc. before LLM inference, but does not strip:

• <aside> (semantic HTML5 sidebar tag)
• Elements with sidebar-related IDs (id="recommend", id="sidebar")
• Elements with ARIA roles (role="complementary", role="navigation")

These survive preprocessing and get _item_id annotations. When the LLM misclassifies them as "main", the noise enters main_html.

The current ATTR_PATTERNS_TO_REMOVE in simplify_html.py only contains 'nav':

ATTR_PATTERNS_TO_REMOVE = {
    'nav',  # 'footer', 'header',  # standalone words
}

Suggestion

Would you consider extending simplify_html.py to also strip <aside> elements (similar to the existing <nav> handling in tags_to_remove)? The <aside> tag has clear semantic meaning in HTML5 — it represents content "tangentially related" to the main content, which aligns with what should be classified as "other".

This would be a conservative extension that follows the same pattern as the existing <nav> removal and reduces the chance of LLM misclassification on sidebar content.

Environment

• Model: opendatalab/MinerU-HTML (v1.0, Qwen3-based)
• Code: commit 73cf2666
• We are planning to test v1.1 (MinerU-HTML-v1.1-hunyuan0.5B-compact) on these cases to see if the improved model accuracy resolves the issue.

Sample Data

We can share the full 240-page CSDN sample (raw HTML + Dripper output + extracted markdown) if helpful for reproducing and testing fixes.

[darkrush]

Thank you very much for providing such a detailed and high-quality usage report! You not only reproduced the specific bad cases, but also included statistical data, Root Cause analysis, and improvement suggestions — this is extremely helpful to us.

1. v1.1 model has resolved the issue
We fully tested the 3 CSDN pages you provided (including sidebar recommendations, GitCode spam links, etc.) using the latest v1.1 version + MinerU-HTML-v1.1-hunyuan0.5B-compact model. The results show no additional junk content leaking into main_html. The new model has effectively overcome these misclassification problems.

If you would like to quickly verify these bad cases on v1.1 yourself, you can:

• Register a free Token at MinerU
• Refer to the HTML Processing API documentation and call it directly.

The API currently uses the exact same implementation, code, and model as our latest open-source version.

2. Explanation of our filtering strategy
We currently only filter a small number of tags (such as <nav>). The main reason is that in our benchmark, a large number of web pages do not strictly follow HTML5 semantics — especially those sampled from CommonCrawl with extremely high noise levels. If we expand the filtering rules further (e.g., removing all <aside> tags), it would easily cause real main content on some pages to be incorrectly filtered out.

Therefore, we have chosen a very conservative filtering strategy, leaving more of the judgment to the large model based on the actual content. This is also why we currently have no plans to significantly adjust ATTR_PATTERNS_TO_REMOVE or tags_to_remove in the official repository.

That said, we warmly welcome community customization:

• If you know in your own use cases which tags/attributes can be safely filtered by rules, you can easily extend the rules directly in simplify_html.py.
• With the help of large models, targeted rule adjustments for specific domains will also become quite straightforward.

If you still encounter any issues after testing v1.1, or if you have more bad cases from other scenarios, please feel free to share them — we will continue to follow up and optimize.

Thank you again for your valuable feedback and contribution! 🚀

---

非常感谢您提供如此详细且高质量的使用报告！您不仅复现了具体 bad case，还附上了统计数据、Root Cause 分析和改进建议，这对我们非常有帮助。

1. v1.1 模型已解决该问题
我们在最新的 v1.1 版本 + MinerU-HTML-v1.1-hunyuan0.5B-compact 模型上，完整测试了您提供的 3 个 CSDN 案例（包括 sidebar 推荐、GitCode 垃圾链接等）。结果显示没有额外的垃圾内容进入 main_html，新的模型已能有效克服这些误分类问题。

如果您想快速验证这些 bad case 在 v1.1 上的实际效果，可以：

• 在 MinerU 注册一个 Token（免费）
• 参考 HTML 处理 API 文档 直接调用
  该 API 当前使用的实现、代码和模型与我们最新开源版本完全一致。

2. 关于过滤策略的说明
我们目前只过滤少量标签（如 <nav>），主要原因是：在我们的 benchmark 中，存在大量网页并未严格遵循 HTML5 语义（尤其是来自 CommonCrawl 的高噪声页面）。如果进一步扩大过滤规则（例如直接移除所有 <aside>），很容易把某些页面的真实正文错误过滤掉。

因此我们选择了非常保守的过滤策略，把更多判断交给大模型根据实际内容来完成。这也是我们目前不打算在官方仓库中大幅调整 ATTR_PATTERNS_TO_REMOVE 和 tags_to_remove 的原因。

不过我们非常欢迎社区的自定义需求：

• 如果您在自己的业务场景里明确知道某些标签/属性可以安全过滤，完全可以在 simplify_html.py 中自行扩展规则（操作非常简单）。
• 借助大模型的帮助，后续针对特定领域的规则微调也会变得很容易。

如果您在测试 v1.1 后仍有任何问题，或者有更多场景的 bad case，欢迎随时贴出来，我们会继续跟进优化。

再次感谢您的反馈和贡献！🚀