Skip to content

fix: Fix some knowledge base import errors #2739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shaohuzhang1 merged 1 commit into from
Mar 31, 2025
Merged

fix: Fix some knowledge base import errors #2739

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions apps/dataset/serializers/common_serializers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,24 @@ def zip_dir(zip_path, output=None):
zip.close()


def is_valid_uuid(s):
try:
uuid.UUID(s)
return True
except ValueError:
return False


def write_image(zip_path: str, image_list: List[str]):
for image in image_list:
search = re.search("\(.*\)", image)
if search:
text = search.group()
if text.startswith('(/api/file/'):
r = text.replace('(/api/file/', '').replace(')', '')
r = r.strip().split(" ")[0]
if not is_valid_uuid(r):
break
file = QuerySet(File).filter(id=r).first()
if file is None:
break
Expand All @@ -58,6 +69,9 @@ def write_image(zip_path: str, image_list: List[str]):
f.write(file.get_byte())
else:
r = text.replace('(/api/image/', '').replace(')', '')
r = r.strip().split(" ")[0]
if not is_valid_uuid(r):
break
image_model = QuerySet(Image).filter(id=r).first()
if image_model is None:
break
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Mar 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided Python code checks image references in a ZIP archive for URL patterns that include UUIDs and validates each one using uuid.UUID. The code handles regular expressions to extract URLs from images' descriptions and filters out those referencing non-valid UUIDs.

Here's a suggestion for improvement:

Add error handling around file operations to manage read/write errors more robustly. For example, handle exceptions during opening files for writing and ensure resources like the ZIP archive are properly closed after use even if an error occurs.

import os
from typing import List

def zip_dir(zip_path: str, output='zip_files'):
    try:
        with zipfile.ZipFile(zip_path, mode="w") as zip_handle:
            # Add your logic here to add files to the zip-archive
    except Exception as e:
        print(f"An error occurred while creating or writing to {zip_path}: {e}")

Additionally, consider adding logging information at different points within the function (like when processing individual items) to help debug potential issues. This can be done using Python’s built-in logging module.

Expand Down
Loading