Skip to content

WIP: mongodb and sql optimizaitons#3101

Open
doomedraven wants to merge 2 commits into
masterfrom
mongo_n_sql
Open

WIP: mongodb and sql optimizaitons#3101
doomedraven wants to merge 2 commits into
masterfrom
mongo_n_sql

Conversation

@doomedraven

Copy link
Copy Markdown
Collaborator

No description provided.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

@doomedraven

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces MongoDB query and regex timeouts, adds a regex length limit to prevent ReDoS, replaces subqueryload with selectinload in SQLAlchemy queries, and refactors transaction handling. However, several critical issues were identified in the review. Specifically, PyMongo's bulk_write, delete_one, delete_many, update_one, and update_many methods do not support the maxTimeMS parameter, which will cause runtime TypeError exceptions. Additionally, refactoring transaction handling to use begin_nested() only creates database savepoints without committing the outer transactions, preventing deletions and session creations from being persisted.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread dev_utils/mongodb.py
Comment on lines 160 to 163
def mongo_bulk_write(collection: str, requests, **kwargs):
if "maxTimeMS" not in kwargs:
kwargs["maxTimeMS"] = query_timeout
return getattr(results_db, collection).bulk_write(requests, **kwargs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

In PyMongo, the bulk_write method does not accept arbitrary keyword arguments or maxTimeMS as a parameter. Passing maxTimeMS will result in a TypeError: bulk_write() got an unexpected keyword argument 'maxTimeMS'. If you need to enforce a timeout on write operations, you should configure it at the client level (e.g., using socketTimeoutMS or the timeout parameter in PyMongo 4+).

def mongo_bulk_write(collection: str, requests, **kwargs):
    return getattr(results_db, collection).bulk_write(requests, **kwargs)

Comment thread dev_utils/mongodb.py Outdated
Comment thread dev_utils/mongodb.py Outdated
Comment on lines +230 to +233
@graceful_auto_reconnect
def mongo_delete_many(collection: str, query):
return getattr(results_db, collection).delete_many(query)
max_time_ms = regex_timeout if is_regex_query(query) else query_timeout
return getattr(results_db, collection).delete_many(query, maxTimeMS=max_time_ms)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

In PyMongo, the delete_many method does not accept maxTimeMS or max_time_ms as a keyword argument. Attempting to pass maxTimeMS will raise a TypeError: delete_many() got an unexpected keyword argument 'maxTimeMS'. Please remove maxTimeMS from the call.

Suggested change
@graceful_auto_reconnect
def mongo_delete_many(collection: str, query):
return getattr(results_db, collection).delete_many(query)
max_time_ms = regex_timeout if is_regex_query(query) else query_timeout
return getattr(results_db, collection).delete_many(query, maxTimeMS=max_time_ms)
def mongo_delete_many(collection: str, query):
return getattr(results_db, collection).delete_many(query)

Comment thread dev_utils/mongodb.py Outdated
Comment thread dev_utils/mongodb.py Outdated
Comment on lines 1143 to 1150
try:
result = self.session.execute(delete_stmt)
log.info("Deleted %d tasks matching the criteria.", result.rowcount)
self.session.commit()
with self.session.begin_nested():
result = self.session.execute(delete_stmt)
log.info("Deleted %d tasks matching the criteria.", result.rowcount)
return True
except SQLAlchemyError as e:
log.error("Error deleting tasks: %s", str(e))
self.session.rollback()
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using with self.session.begin_nested(): only creates a database SAVEPOINT (nested transaction) and does not commit the outer transaction. Since delete_tasks is called from outside any active transaction context in several places (such as cleaners_utils.py), the deletion will never be committed to the database and will be rolled back when the session is closed. Please revert to using explicit self.session.commit() and self.session.rollback() to ensure the transaction is committed correctly.

Suggested change
try:
result = self.session.execute(delete_stmt)
log.info("Deleted %d tasks matching the criteria.", result.rowcount)
self.session.commit()
with self.session.begin_nested():
result = self.session.execute(delete_stmt)
log.info("Deleted %d tasks matching the criteria.", result.rowcount)
return True
except SQLAlchemyError as e:
log.error("Error deleting tasks: %s", str(e))
self.session.rollback()
return False
try:
result = self.session.execute(delete_stmt)
self.session.commit()
log.info("Deleted %d tasks matching the criteria.", result.rowcount)
return True
except SQLAlchemyError as e:
self.session.rollback()
log.error("Error deleting tasks: %s", str(e))
return False

Comment on lines 229 to 235
try:
guac = GuacSession(token=str(token), task_id=task_id, vm_label=vm_label, guest_ip=guest_ip)
session.add(guac)
session.commit()
with self.session.begin_nested():
guac = GuacSession(token=str(token), task_id=task_id, vm_label=vm_label, guest_ip=guest_ip)
self.session.add(guac)
return guac
except Exception:
session.rollback()
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using with self.session.begin_nested(): only creates a database SAVEPOINT and does not commit the outer transaction. Since create_guac_session is called from outside any active transaction context, the newly created GuacSession will never be committed to the database. Please use explicit self.session.commit() and self.session.rollback() to ensure the session is saved.

Suggested change
try:
guac = GuacSession(token=str(token), task_id=task_id, vm_label=vm_label, guest_ip=guest_ip)
session.add(guac)
session.commit()
with self.session.begin_nested():
guac = GuacSession(token=str(token), task_id=task_id, vm_label=vm_label, guest_ip=guest_ip)
self.session.add(guac)
return guac
except Exception:
session.rollback()
raise
try:
guac = GuacSession(token=str(token), task_id=task_id, vm_label=vm_label, guest_ip=guest_ip)
self.session.add(guac)
self.session.commit()
return guac
except Exception:
self.session.rollback()
raise

@doomedraven doomedraven changed the title mongodb and sql optimizaitons WIP: mongodb and sql optimizaitons Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant