Skip to content

fix: Apply SQLite optimizations to the custom connection_string in SqlStorageClient #1837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vdusek merged 3 commits into from
Apr 7, 2026
Merged

fix: Apply SQLite optimizations to the custom connection_string in SqlStorageClient #1837

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
88eba6e
Apply SQLite optimizations to the custom `connection_string` in `SqlS…
Mantisus Apr 5, 2026
9d31eb2
add tests
Mantisus Apr 6, 2026
1c6c264
set pragma per connect
Mantisus Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions src/crawlee/storage_clients/_sql/_storage_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,10 @@ def __init__(
self._initialized = False
self.session_maker: None | async_sessionmaker[AsyncSession] = None

# Flag needed to apply optimizations only for default database
self._default_flag = self._engine is None and self._connection_string is None
# Flag needed to apply optimizations only for SQLite database
self._optimization_flag = self._engine is None and (
Copy link
Copy Markdown
Collaborator

@janbuchar janbuchar Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we instead set up an _on_connect callback here that would receive the new DB connection? That way, the logic (if sqlite, set up some pragmas) would be in one place.

Copy link
Copy Markdown
Collaborator Author

@Mantisus Mantisus Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

self._connection_string is None or self._connection_string.startswith('sqlite')
)
Comment thread
Mantisus marked this conversation as resolved.
Outdated
self._dialect_name: str | None = None

# Call the notification only once
Expand Down Expand Up @@ -129,7 +131,7 @@ async def initialize(self, configuration: Configuration) -> None:
# This is likely an attempt to create a database from several parallel processes.
try:
# Set SQLite pragmas for performance and consistency
if self._default_flag:
if self._optimization_flag:
await conn.execute(text('PRAGMA journal_mode=WAL')) # Better concurrency
await conn.execute(text('PRAGMA synchronous=NORMAL')) # Balanced safety/speed
await conn.execute(text('PRAGMA cache_size=100000')) # 100MB cache
Expand Down
4 changes: 4 additions & 0 deletions uv.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading