Skip to content

fix: Some web pages are unable to be crawled #3897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shaohuzhang1 merged 1 commit into from
Aug 20, 2025
Merged

fix: Some web pages are unable to be crawled #3897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions apps/common/util/fork.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,28 @@ def remove_fragment(url: str) -> str:
return urlunparse(modified_url)


def remove_last_path_robust(url):
"""健壮地删除URL的最后一个路径部分"""
parsed = urlparse(url)

# 分割路径并过滤空字符串
paths = [p for p in parsed.path.split('/') if p]

if paths:
paths.pop() # 移除最后一个路径

# 重建路径
new_path = '/' + '/'.join(paths) if paths else '/'

# 重建URL
return urlunparse((
parsed.scheme,
parsed.netloc,
new_path,
parsed.params,
parsed.query,
parsed.fragment
))
class Fork:
class Response:
def __init__(self, content: str, child_link_list: List[ChildLink], status, message: str):
Expand All @@ -72,7 +94,7 @@ def error(message: str):
def __init__(self, base_fork_url: str, selector_list: List[str]):
base_fork_url = remove_fragment(base_fork_url)
if any([True for end_str in ['index.html', '.htm', '.html'] if base_fork_url.endswith(end_str)]):
self.base_fork_url = str(Path(base_fork_url).parent)
base_fork_url =remove_last_path_robust(base_fork_url)
self.base_fork_url = urljoin(base_fork_url if base_fork_url.endswith("/") else base_fork_url + '/', '.')
parsed = urlsplit(base_fork_url)
query = parsed.query
Expand Down Expand Up @@ -190,4 +212,4 @@ def fork(self):
def handler(base_url, response: Fork.Response):
print(base_url.url, base_url.tag.text if base_url.tag else None, response.content)

# ForkManage('https://bbs.fit2cloud.com/c/de/6', ['.md-content']).fork(3, set(), handler)
# ForkManage('https://hzqcgc.htc.edu.cn/jxky.htm', ['.md-content']).fork(3, set(), handler)
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code includes several improvements to enhance robustness and readability:

  1. Function remove_last_path_robust: This function processes URLs by splitting them into path components, removing the last one, and then reconstructing the URL. It handles empty paths gracefully.

  2. Error Handling in Class Fork.Manage:

    • The constructor now uses the corrected URL obtained from remove_last_path_robust instead of directly modifying it.
    • Added checks within the constructor to handle different endings (e.g., .html, .htm) before setting the base fork URL.

  3. Code Structure Optimization:

    • The class methods maintain a consistent structure with clear initialization, parameter handling (base_url and response). This improves code clarity and reusability.
    • Minor refactoring is done to make certain lines more concise where appropriate.

These changes improve the overall quality and reliability of the code.

3 changes: 3 additions & 0 deletions ui/src/locales/lang/en-US/ai-chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ export default {
limitMessage2: 'files',
sizeLimit: 'Each file must not exceed',
imageMessage: 'Please process the image content',
documentMessage: 'Please understand the content of the document',
audioMessage: 'Please understand the video content',
otherMessage: 'Please understand the file content',
errorMessage: 'Upload Failed'
},
executionDetails: {
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code snippet you provided seems to be a part of an object definition within a JavaScript module exported from ES6/ESNext syntax. It defines several properties for error messages related to various input types (e.g., files, images, documents). Here are some points to consider:

Irregularities:

  • No explicit export statement is used above or below this object declaration. This might lead to confusion if the module exports multiple items.
  • The file extension .js suggests that this is a Node.js file.

Potential Issues:

  1. Lack of Import Export: While not strictly incorrect, missing import/export statements can make it harder to reuse or extend this code without modifying source files directly.
  2. Empty Object for Execution Details: The executionDetails property is assigned as an empty object ({}), which doesn't appear to have been filled yet. Depending on how this module will be structured, you might want to initialize it with defaults or define it elsewhere.

Optimization Suggestions:

  1. Consistent Formatting: Ensure there's consistent spacing (either all tabs or all spaces) throughout the file to improve readability.
  2. Consider Using Typescript: If your project uses TypeScript, converting this object into a type would help catch potential errors at compile time rather than runtime.

Here’s a slightly improved version of the structure, adding appropriate comments and ensuring consistency:

// Define message strings for different input types
const uploadErrors = {
  limitMessageFile: 'files',
  sizeLimit: 'Each file must not exceed',
  imageMessage: 'Please process the image content',
  documentMessage: 'Please understand the content of the document',
  audioMessage: 'Please understand the video content',
  otherMessage: 'Please understand the file content',
  errorMessage: 'Upload Failed'
};

// Additional details object for future use or updates
const executionDetails = {};

export { uploadErrors, executionDetails };

This version includes comments explaining each component and properly initializes the executionDetails object. Adjusting import/export as needed based on your specific use case further enhances modularity and reusability.

Expand Down
3 changes: 3 additions & 0 deletions ui/src/locales/lang/zh-CN/ai-chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ export default {
limitMessage2: '个文件',
sizeLimit: '单个文件大小不能超过',
imageMessage: '请解析图片内容',
documentMessage: '请理解文档内容',
audioMessage: '请理解视频内容',
otherMessage: '请理解文件内容',
errorMessage: '上传失败'
},
executionDetails: {
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code looks mostly correct regarding syntax, but there are a few improvements that can be made:

  1. Spacing: Ensure consistent spacing around operators like = and : to improve readability.

  2. Comments: Add comments where needed to explain complex logic or sections of the code for future maintainers.

  3. Variable Naming Consistency: Maintain consistency in variable naming style (e.g., camelCase).

  4. Documentation Strings: Consider adding more comprehensive docstrings if applicable.

Here’s an optimized version of your code with these suggestions:

// Define translations messages
export default {
  limitMessage1: '{count} 文件',
  limitMessage2: '个文件', // Assuming count is already defined elsewhere
  sizeLimit: '单个文件大小不能超过 {size}',
  imageMessage: '请解析图片内容。',
  documentMessage: '请理解文档内容。', // New translation for documents
  audioMessage: '请理解音频内容。',   // New translation for videos
  otherMessage: '请理解文件内容。'    // New translation for all types of files except images, documents, and audio
};

// Provide additional details about the execution process
executionDetails: {
  /*
    This object contains information about the execution flow.
    It might include steps performed during execution,
    error codes associated with failed tasks, etc.
  */
}

Additional Suggestions:

  • Error Handling: Ensure proper handling of API errors, file size limits, and invalid file formats.

  • Dynamic Values: If {count} needs to be dynamically updated, ensure it's passed correctly.

  • Translational Context: Ensure each message is clear and relevant to its context.

These changes should make the code both functional and easy to maintain.

Expand Down
3 changes: 3 additions & 0 deletions ui/src/locales/lang/zh-Hant/ai-chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ export default {
limitMessage2: '個文件',
sizeLimit: '單個文件大小不能超過',
imageMessage: '請解析圖片內容',
documentMessage: '請理解檔案內容',
audioMessage: '請理解視頻內容',
otherMessage: '請理解檔案內容',
errorMessage: '上傳失敗'
},
executionDetails: {
Expand Down
Loading