Skip to content

fix: psycopg3 connection issues on windows debugging environment.#47

Merged
beersoccer merged 1 commit into
beersoccer:mainfrom
sususweet:fix/psycopg-connection-issue
May 16, 2026
Merged

fix: psycopg3 connection issues on windows debugging environment.#47
beersoccer merged 1 commit into
beersoccer:mainfrom
sususweet:fix/psycopg-connection-issue

Conversation

@sususweet
Copy link
Copy Markdown

When this plugin is running or debugging on Windows platform, it may encounter psycopg3 connection issues, as the error log shown below:

"sending query failed: another command is already in progress, another command is already in progress"

{"event": "log", "data": {"level": "INFO", "message": "Created psycopg3 ConnectionPool: min_size=1, max_size=10, max_lifetime=3600.0, max_idle=600.0, timeout=30.0, reconnect_timeout=300.0. Connection string: postgresql://postgres:***@192.168.73.128:8432/postgres?sslmode=disable&keepalives=1&keepalives_idle=30&keepalives_interval=10&keepalives_count=3&connect_timeout=5", "timestamp": 1776914680.0933197}}
{"event": "log", "data": {"level": "ERROR", "message": "Credential validation failed", "timestamp": 1776914682.5841715}}
WARNING:psycopg:error ignored in rollback on <psycopg.Connection [ACTIVE] (host=192.168.73.128 port=8432 database=postgres) at 0x285cd4e5640>: sending query failed: another command is already in progress
another command is already in progress
WARNING:psycopg.pool:closing returned connection: <psycopg.Connection [ACTIVE] (host=192.168.73.128 port=8432 database=postgres) at 0x285cd4e5640>
WARNING:psycopg.pool:discarding closed connection: <psycopg.Connection [BAD] at 0x285cd4e5640>
ERROR:dify_plugin.core.server.io_server:Unexpected error occurred when executing request
Traceback (most recent call last):
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\vector_stores\pgvector.py", line 124, in _get_cursor
    yield cur
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\vector_stores\pgvector.py", line 320, in list_cols
    cur.execute("SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'")
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\cursor.py", line 113, in execute
    self._conn.wait(
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\connection.py", line 483, in wait
    return waiting.wait(gen, self.pgconn.socket, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\waiting.py", line 85, in wait_selector
    sel.register(fileno, s)
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\gevent\selectors.py", line 146, in register
    key = _BaseSelectorImpl.register(self, fileobj, events, data)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python312\Lib\selectors.py", line 241, in register
    raise KeyError("{!r} (FD {}) is already registered"
KeyError: '1264 (FD 1264) is already registered'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\code\mem0_dify_plugin\provider\mem0ai.py", line 95, in _validate_credentials
    client = get_sync_client(credentials)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\utils\mem0_client.py", line 1722, in get_sync_client
    _cache["sync_client"] = SyncMem0Client(credentials)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\utils\mem0_client.py", line 307, in __init__
    self.memory = Memory.from_config(config)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\memory\main.py", line 243, in from_config
    return cls(config)
           ^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\memory\main.py", line 183, in __init__
    self.vector_store = VectorStoreFactory.create(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\utils\factory.py", line 198, in create
    return vector_store_instance(**config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\vector_stores\pgvector.py", line 109, in __init__
    collections = self.list_cols()
                  ^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\vector_stores\pgvector.py", line 319, in list_cols
    with self._get_cursor() as cur:
         ^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python312\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\mem0\vector_stores\pgvector.py", line 128, in _get_cursor
    conn.rollback()
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\connection.py", line 309, in rollback
    self.wait(self._rollback_gen())
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\connection.py", line 483, in wait
    return waiting.wait(gen, self.pgconn.socket, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\waiting.py", line 73, in wait_selector
    s = next(gen)
        ^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\_connection_base.py", line 610, in _rollback_gen
    yield from self._exec_command(b"ROLLBACK")
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\psycopg\_connection_base.py", line 476, in _exec_command
    self.pgconn.send_query(command)
  File "psycopg_binary/pq/pgconn.pyx", line 310, in psycopg_binary.pq.PGconn.send_query
psycopg.OperationalError: sending query failed: another command is already in progress

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\dify_plugin\core\server\io_server.py", line 94, in _execute_request_in_thread
    self._execute_request(
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\dify_plugin\plugin.py", line 468, in _execute_request
    response = self.dispatch(session, data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\dify_plugin\core\server\router.py", line 79, in dispatch
    return route.func(session, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\dify_plugin\core\server\router.py", line 72, in wrapper
    return f(session, data)
           ^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\dify_plugin\core\plugin_executor.py", line 96, in validate_tool_provider_credentials
    provider_instance.validate_credentials(data.credentials)
  File "E:\code\mem0_dify_plugin\.venv\Lib\site-packages\dify_plugin\interfaces\tool\__init__.py", line 235, in validate_credentials
    return self._validate_credentials(credentials)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\mem0_dify_plugin\provider\mem0ai.py", line 127, in _validate_credentials
    raise ToolProviderCredentialValidationError(str(e)) from e
dify_plugin.errors.tool.ToolProviderCredentialValidationError: sending query failed: another command is already in progress

So this PR changes the WRITE_OPERATION_TIMEOUT to avoid slow database I/O and uses fallback to psycopg2 on Windows platform to enable plugin running and debugging on Windows.

@beersoccer
Copy link
Copy Markdown
Owner

Thanks for the detailed diagnosis and the Windows reproduction steps — the root cause identification is spot on.

After reviewing the three changes, here's my assessment:


main.py — event loop policy fix

This correctly addresses the root cause (gevent selector conflict with Windows ProactorEventLoop). Happy to merge this part.


WRITE_OPERATION_TIMEOUT 15 → 120

The timeout issue on Windows is a symptom of the selector conflict, not slow database I/O. Increasing this to 120s won't fix the underlying bug — it just delays the error surfacing. More importantly, WRITE_OPERATION_TIMEOUT is intentionally kept below MAX_REQUEST_TIMEOUT (60s); going to 120s would exceed the request boundary and cause sync-mode users to wait up to 2 minutes before seeing any error. Please revert this change.


requirements.txt / pyproject.toml — replacing psycopg3 with psycopg2 on Windows

Removing psycopg3 on Windows would break pgvector entirely, because mem0ai's internal pgvector implementation directly imports psycopg (psycopg3). Without it, Windows users would hit an ImportError instead — trading one error for another.

That said, the project already has psycopg2 fallback logic in pgvector_config.py. If you'd like to make that fallback available on Windows, the correct approach is to keep psycopg3 and additionally install psycopg2, rather than replacing it:

psycopg[binary,pool]                           # all platforms (mem0 requires it)
psycopg2-binary ; sys_platform == "win32"      # Windows extra fallback

Could you update the PR to keep only the event loop policy change (and optionally the corrected dependency approach above)? That would be ready to merge.

@sususweet sususweet force-pushed the fix/psycopg-connection-issue branch from 463230c to 03b6980 Compare April 29, 2026 06:52
@sususweet
Copy link
Copy Markdown
Author

Thanks for your detailed review of this PR. After testing and debugging on the Windows platform, we identified several issues that may need further discussion to clarify the proposed changes.


WRITE_OPERATION_TIMEOUT: 15 → 45

Through repeated tests of reading from and writing to the pgvector database, we observed that the read process is significantly faster than the write process. This implies that WRITE_OPERATION_TIMEOUT should be set slightly higher than READ_OPERATION_TIMEOUT; otherwise, memory writes may frequently time out. We have therefore increased this constant to 45 seconds to make the write process more tolerant. Could you please review whether this change is acceptable?


requirements.txt / pyproject.toml — replacing psycopg3 with psycopg2 on Windows

We modified requirements.txt as shown below and re‑debugged the plugin on Windows:

mem0ai>=1.0.2,<=1.0.11
openai
azure-identity
langchain-neo4j
rank-bm25
psycopg[binary,pool]
psycopg2-binary      ; sys_platform == "win32"  # psycopg2 pool (Windows only)
dify_plugin
tiktoken  # Accurate token counting for conversation processing
cohere>=6.1.0

However, the mem0 library still attempts to use psycopg3 (the newer driver) because its driver selection logic in vector_stores/pgvector.py follows this order (see screenshot):

image
# Try to import psycopg (psycopg3) first, then fall back to psycopg2
try:
    from psycopg.types.json import Json
    from psycopg_pool import ConnectionPool
    PSYCOPG_VERSION = 3
    logger = logging.getLogger(__name__)
    logger.info("Using psycopg (psycopg3) with ConnectionPool for PostgreSQL connections")
except ImportError:
    try:
        from psycopg2.extras import Json, execute_values
        from psycopg2.pool import ThreadedConnectionPool as ConnectionPool
        PSYCOPG_VERSION = 2
        logger = logging.getLogger(__name__)
        logger.info("Using psycopg2 with ThreadedConnectionPool for PostgreSQL connections")
    except ImportError:
        raise ImportError(
            "Neither 'psycopg' nor 'psycopg2' library is available. "
            "Please install one of them using 'pip install psycopg[pool]' or 'pip install psycopg2'"
        )

Therefore, on Windows, to force mem0 to use psycopg2 as the connection driver, the psycopg (i.e., psycopg3) library must not be installed.


Hope you could review this PR and see whether it could be merged.

@sususweet
Copy link
Copy Markdown
Author

Hope you could take some time to review this PR and see whether it could be merged.

@beersoccer
Copy link
Copy Markdown
Owner

Hope you could take some time to review this PR and see whether it could be merged.

Thanks for your patience and for taking the time to revise the PR based on my feedback — and sorry for the slow review on my end.

After a closer look at the updated diff and verifying that mem0ai does not list psycopg as a required dependency (confirmed via pip show mem0ai), I'm satisfied that all three changes are correct and safe to merge:

main.py — event loop policy: Cleanly scoped to win32, zero impact on Linux/macOS production deployments.
pyproject.toml / requirements.txt — conditional psycopg install: The sys_platform != 'win32' guard is exactly the right approach. Since mem0ai doesn't pull in psycopg3 transitively, excluding it on Windows will correctly trigger mem0's built-in ImportError fallback to psycopg2. The existing psycopg2 pool path in pgvector_config.py handles this transparently.
WRITE_OPERATION_TIMEOUT 15 → 45: Your point about writes being inherently slower than reads (extra LLM inference step) is valid. 45 s is well within MAX_REQUEST_TIMEOUT (60 s), and reducing silent write timeouts in async mode is a genuine reliability improvement.
I apologize for the delayed response — I should have reviewed this more promptly. Going to merge this now. Thanks again for the detailed diagnosis and the Windows reproduction steps!

@beersoccer beersoccer merged commit 3ff3a81 into beersoccer:main May 16, 2026
4 checks passed
@sususweet sususweet deleted the fix/psycopg-connection-issue branch May 18, 2026 00:55
@sususweet
Copy link
Copy Markdown
Author

Thank you for the thorough review and thoughtful feedback! I really appreciate you taking the time to re-examine the changes and explain the reasoning behind each one — especially the confirmation that mem0ai doesn't depend on psycopg transitively.

I'm glad the three changes make sense and are safe to merge:

  • The event loop policy scoped to Windows avoids any impact on Linux/macOS.
  • The conditional psycopg install with sys_platform != 'win32' is a clean way to let mem0 fall back to psycopg2 on Windows as intended.
  • Increasing WRITE_OPERATION_TIMEOUT to 45s should help with the inherent slowdown of writes without hitting the 60s max.

No worries at all about the review timing — I completely understand how busy things get. Thanks again for your patience and for merging the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants