Replace databases with sqlalchemy async (#1515)

* Replace databases library with SQLAlchemy async API

Migrate the project from the databases library to using SQLAlchemy's native async API directly.

- ormar/connection.py: DatabaseConnection class that wraps AsyncEngine
- ormar/query_executor.py: QueryExecutor class for database operations
- ormar/transaction.py: Transaction management with savepoint support

- Updated OrmarConfig, QuerySet, Model, and related files to use new async patterns
- Simplified SQLite pragma handling
- Updated all dialect access patterns

- Updated test files to use DatabaseConnection and async drivers
- Changed DDL operations to use async engine with run_sync()

* Fix connection management and transaction handling

Fixes critical connection pool exhaustion and transaction handling issues.

Key changes:
- Removed auto-commit from QueryExecutor
- Use engine.begin() for auto-commit connections
- Added connection pool limits (pool_size=5, max_overflow=10)
- Fixed transaction connection reuse

* fix all tests to pass

* fix all tests to pass in all backends

* update gitignore

* add no cover to sqlite test in lifespan for other engines related part

* quick and ugly docs src fix to pass test docs workflow

* remove unused substitue_backend_pool_for_sqlite and duplicate code in model.py

* simplify getting the query executor to avoid code duplication

* simplify prefetch related query

* add back force rollback to signals that was not working with databases, remove unused cleanup, double check force rollback on db without transaction is working too

* add more tests for transactions, ensure sqlite transaction rollback with nested transactions

* fix depth check and change to value error to avoid swallowing exceptions in transactions tests

* remove engine from docs sources as its optional and we handle db creation and drop separately

* fix example too

* updates the lines references after the docs changes and fixes

* updates the documentation to reflect the new DatabaseConnection dependency and migration to async SQLAlchemy. Add migration example and update dependencies for async support.

* add relases and bump version to 0.22.0

* bump version to 0.22.0

* bump poetry version in workflows

* fix benchmarks not connected error

* bump pytest-asyncio version to 1.2.0 and update benchmarks to not use event loop fixture

* bump pytest-cov

* restore benchmark fixture decorator

* revert pytest-asyncio to check pipeline

* check 1.2 version of pytest-asyncio

* Revert "check 1.2 version of pytest-asyncio"

This reverts commit e964f44.

* Reapply "check 1.2 version of pytest-asyncio"

This reverts commit 437b956.

* check if new fixture is messing up the coverage

* Revert "check if new fixture is messing up the coverage"

This reverts commit 0353e49.

* Revert "Reapply "check 1.2 version of pytest-asyncio""

This reverts commit e71cd75.

* update benchmark mode to simulation and python 3.12

* update benchmarks remove options

* update benchmarks restore part of the options

* update benchmarks restore part of the options

* Revert "update benchmarks restore part of the options"

This reverts commit 4917ebc.

* Revert "update benchmarks restore part of the options"

This reverts commit 2fe38dc.

* Revert "update benchmarks remove options"

This reverts commit 5fa3950.

* Revert "update benchmark mode to simulation and python 3.12"

This reverts commit 80e7ff3.

* update benchmarks restore part of the options

* run on codspeed-macro instead of ubuntu-latest

* revert to ubuntu-latest for benchmarks and instrumentation mode

* avoid begin if not in transaction, asyncio sleep every 100 rows for better performance

Author: collerek

.github/workflows/benchmark.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Install Poetry
         uses: snok/install-poetry@v1.4
         with:
-          version: 1.8.4
+          version: 2.1.3
           virtualenvs-create: false
 
       - name: Poetry details

.github/workflows/deploy-docs.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Install Poetry
         uses: snok/install-poetry@v1.4
         with:
-          version: 1.8.4
+          version: 2.1.3
           virtualenvs-create: false
       - name: Install dependencies
         run: |

.github/workflows/lint.yml

@@ -26,7 +26,7 @@ jobs:
     - name: Install Poetry
       uses: snok/install-poetry@v1.4
       with:
-        version: 1.8.4
+        version: 2.1.3
         virtualenvs-create: false
 
     - name: Poetry details

.github/workflows/python-publish.yml

@@ -25,7 +25,7 @@ jobs:
     - name: Install Poetry
       uses: snok/install-poetry@v1.4
       with:
-        version: 1.8.4
+        version: 2.1.3
         virtualenvs-create: true
         virtualenvs-in-project: true
 

.github/workflows/test-package.yml

@@ -54,7 +54,7 @@ jobs:
     - name: Install Poetry
       uses: snok/install-poetry@v1.4
       with:
-        version: 1.8.4
+        version: 2.1.3
         virtualenvs-create: false
 
     - name: Poetry details

.github/workflows/test_docs.yml

@@ -24,7 +24,7 @@ jobs:
     - name: Install Poetry
       uses: snok/install-poetry@v1.4
       with:
-        version: 1.8.4
+        version: 2.1.3
         virtualenvs-create: false
     - name: Install dependencies
       run: |

.github/workflows/type-check.yml

@@ -26,7 +26,7 @@ jobs:
     - name: Install Poetry
       uses: snok/install-poetry@v1.4
       with:
-        version: 1.8.4
+        version: 2.1.3
         virtualenvs-create: false
 
     - name: Poetry details

.gitignore

@@ -18,3 +18,4 @@ profile.py
 *.db-journal
 *coverage.xml
 .benchmarks/
+CLAUDE.md

benchmarks/conftest.py

@@ -50,6 +50,17 @@ class Book(ormar.Model):
 create_test_database = init_tests(base_ormar_config, scope="function")
 
 
+@pytest_asyncio.fixture(autouse=True, scope="function")
+async def connect_database(create_test_database):
+    if not base_ormar_config.database.is_connected:
+        await base_ormar_config.database.connect()
+
+    yield
+
+    if base_ormar_config.database.is_connected:
+        await base_ormar_config.database.disconnect()
+
+
 @pytest_asyncio.fixture
 async def author():
     author = await Author(name="Author", score=10).save()
@@ -79,14 +90,16 @@ async def authors_in_db(num_models: int):
 @pytest.mark.benchmark(
     min_rounds=1, timer=time.process_time, disable_gc=True, warmup=False
 )
-async def aio_benchmark(benchmark, event_loop: asyncio.BaseEventLoop):
+async def aio_benchmark(benchmark):
     def _fixture_wrapper(func):
         def _func_wrapper(*args, **kwargs):
             if asyncio.iscoroutinefunction(func):
+                # Get the running event loop instead of requesting it as a fixture
+                loop = asyncio.get_running_loop()
 
                 @benchmark
                 def benchmarked_func():
-                    a = event_loop.run_until_complete(func(*args, **kwargs))
+                    a = loop.run_until_complete(func(*args, **kwargs))
                     return a
 
                 return benchmarked_func

docs/fastapi/index.md

@@ -31,11 +31,11 @@ application.
 ```python
 from typing import List, Optional, AsyncIterator
 
-import databases
 import sqlalchemy
 from fastapi import FastAPI
 
 import ormar
+from ormar import DatabaseConnection
 
 from contextlib import asynccontextmanager
 from fastapi import FastAPI
@@ -55,7 +55,7 @@ def get_lifespan(config):
 
 base_ormar_config = ormar.OrmarConfig(
     metadata=sqlalchemy.MetaData(),
-    database=databases.Database("sqlite:///test.db"),
+    database=DatabaseConnection("sqlite+aiosqlite:///test.db"),
 )
 
 app = FastAPI(lifespan=get_lifespan(base_ormar_config))