Skip to content

Commit 22da1b4

Browse files
ericburchamericburcham
committed
Complete Python DataJam implementation with CI/CD pipelines
- Add comprehensive GitHub Actions workflows for both .NET and Python builds - Update README with detailed usage examples and Oracle support documentation - Fix SQLAlchemy 2.0 compatibility by using DeclarativeBase instead of deprecated declarative_base - Clean up unused imports in Oracle container manager - All linting and type checking now passes
1 parent 1ff7187 commit 22da1b4

5 files changed

Lines changed: 313 additions & 35 deletions

File tree

‎.github/workflows/dotnet-build.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Manual .NET build workflow with path filtering
2+
# This supplements the auto-generated nuke-build.yml with path-specific triggers
3+
4+
name: .NET DataJam Build
5+
6+
on:
7+
push:
8+
paths:
9+
- 'src/**'
10+
- 'build/**'
11+
- '*.sln'
12+
- '*.props'
13+
- '*.targets'
14+
- 'global.json'
15+
- '.github/workflows/dotnet-build.yml'
16+
branches:
17+
- main
18+
- develop
19+
- 'feature/**'
20+
pull_request:
21+
paths:
22+
- 'src/**'
23+
- 'build/**'
24+
- '*.sln'
25+
- '*.props'
26+
- '*.targets'
27+
- 'global.json'
28+
- '.github/workflows/dotnet-build.yml'
29+
30+
permissions:
31+
contents: read
32+
actions: write
33+
34+
jobs:
35+
dotnet-build:
36+
name: .NET Build and Test
37+
runs-on: ubuntu-latest
38+
steps:
39+
- name: Checkout code
40+
uses: actions/checkout@v4
41+
with:
42+
fetch-depth: 0
43+
44+
- name: Cache .nuke/temp and NuGet packages
45+
uses: actions/cache@v4
46+
with:
47+
path: |
48+
.nuke/temp
49+
~/.nuget/packages
50+
key: ${{ runner.os }}-${{ hashFiles('**/global.json', '**/*.csproj', '**/Directory.Packages.props') }}
51+
52+
- name: Install .NET 8 SDK
53+
uses: actions/setup-dotnet@v4.1.0
54+
with:
55+
dotnet-version: 8.0.x
56+
dotnet-quality: ga
57+
58+
- name: Install .NET 9 SDK
59+
uses: actions/setup-dotnet@v4.1.0
60+
with:
61+
dotnet-version: 9.0.x
62+
dotnet-quality: ga
63+
64+
- name: Run NUKE build
65+
run: ./build.cmd Nuke
66+
env:
67+
NuGetApiKey: ${{ secrets.NUGET_API_KEY }}
68+
69+
- name: Upload artifacts
70+
uses: actions/upload-artifact@v4
71+
with:
72+
name: dotnet-artifacts
73+
path: artifacts

‎.github/workflows/python-build.yml‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
name: Python DataJam Build
2+
3+
on:
4+
push:
5+
paths:
6+
- 'python/**'
7+
- '.github/workflows/python-build.yml'
8+
pull_request:
9+
paths:
10+
- 'python/**'
11+
- '.github/workflows/python-build.yml'
12+
13+
permissions:
14+
contents: read
15+
actions: write
16+
17+
jobs:
18+
python-build:
19+
name: Python Build and Test
20+
runs-on: ubuntu-latest
21+
strategy:
22+
matrix:
23+
python-version: ["3.10", "3.11", "3.12"]
24+
25+
services:
26+
docker:
27+
image: docker:dind
28+
options: --privileged
29+
30+
steps:
31+
- name: Checkout code
32+
uses: actions/checkout@v4
33+
with:
34+
fetch-depth: 0
35+
36+
- name: Set up Python ${{ matrix.python-version }}
37+
uses: actions/setup-python@v4
38+
with:
39+
python-version: ${{ matrix.python-version }}
40+
41+
- name: Cache Python dependencies
42+
uses: actions/cache@v4
43+
with:
44+
path: |
45+
~/.cache/pip
46+
python/.nox
47+
key: ${{ runner.os }}-python-${{ matrix.python-version }}-${{ hashFiles('python/pyproject.toml', 'python/noxfile.py') }}
48+
restore-keys: |
49+
${{ runner.os }}-python-${{ matrix.python-version }}-
50+
51+
- name: Install Nox
52+
run: |
53+
python -m pip install --upgrade pip
54+
python -m pip install nox
55+
56+
- name: Run linting
57+
working-directory: python
58+
run: nox -s lint
59+
60+
- name: Run unit tests
61+
working-directory: python
62+
run: nox -s test
63+
64+
- name: Run integration tests
65+
working-directory: python
66+
run: nox -s test-integration
67+
env:
68+
TESTCONTAINERS_ORACLE_PASSWORD: oracle123
69+
70+
- name: Build packages
71+
working-directory: python
72+
run: nox -s build
73+
74+
- name: Upload test coverage
75+
if: matrix.python-version == '3.11'
76+
uses: codecov/codecov-action@v3
77+
with:
78+
file: python/coverage.xml
79+
flags: python
80+
name: python-coverage
81+
82+
- name: Upload build artifacts
83+
if: matrix.python-version == '3.11'
84+
uses: actions/upload-artifact@v4
85+
with:
86+
name: python-packages
87+
path: python/artifacts/
88+
89+
publish:
90+
name: Publish Python packages
91+
runs-on: ubuntu-latest
92+
needs: python-build
93+
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
94+
environment: pypi
95+
permissions:
96+
id-token: write # For trusted publishing to PyPI
97+
98+
steps:
99+
- name: Download build artifacts
100+
uses: actions/download-artifact@v4
101+
with:
102+
name: python-packages
103+
path: dist/
104+
105+
- name: Publish to PyPI
106+
uses: pypa/gh-action-pypi-publish@release/v1
107+
with:
108+
packages-dir: dist/

‎python/README.md‎

Lines changed: 124 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,26 +10,65 @@ DataJam provides abstraction patterns for data access across different ORM techn
1010
- **Repository Pattern**: Abstract data access through repository interfaces
1111
- **Unit of Work Pattern**: Manage transaction boundaries and change tracking
1212
- **Oracle Support**: First-class support for Oracle databases with proper case sensitivity handling
13+
- **Async/Await Support**: Modern Python async patterns throughout the stack
14+
- **Type Safety**: Full type hints and mypy compatibility
1315

1416
## Quick Start
1517

1618
```python
17-
from datajam import IRepository, IDataContext
18-
from datajam_sqlalchemy import DataContext, Domain
19-
20-
# Define your domain
21-
class MyDomain(Domain):
22-
def configure_mappings(self, metadata):
23-
# Configure your entity mappings
24-
pass
25-
26-
# Use the repository
27-
domain = MyDomain(connection_string="your_connection_string")
28-
with DataContext(domain) as context:
29-
repository = context.repository
30-
31-
# Execute commands and queries
32-
result = repository.find(MyQuery())
19+
import asyncio
20+
from sqlalchemy.ext.asyncio import create_async_engine
21+
from datajam_sqlalchemy import Domain, DataContext
22+
from datajam import IQuery
23+
24+
# Define your entities using SQLAlchemy
25+
from sqlalchemy import Column, Integer, String
26+
from sqlalchemy.ext.declarative import declarative_base
27+
28+
Base = declarative_base()
29+
30+
class User(Base):
31+
__tablename__ = 'users'
32+
id = Column(Integer, primary_key=True)
33+
name = Column(String(50))
34+
35+
# Create your domain
36+
class UserDomain(Domain):
37+
def __init__(self, engine):
38+
super().__init__(
39+
connection_string=str(engine.url),
40+
engine=engine,
41+
metadata=Base.metadata
42+
)
43+
44+
# Define queries
45+
class GetAllUsers(IQuery[User]):
46+
async def execute(self, data_source):
47+
queryable = data_source.create_query(User)
48+
return await queryable.to_list()
49+
50+
# Usage
51+
async def main():
52+
engine = create_async_engine("oracle+oracledb_async://user:pass@host/service")
53+
domain = UserDomain(engine)
54+
55+
# Create schema
56+
await domain.create_tables()
57+
58+
# Use repository
59+
async with await domain.create_data_context() as context:
60+
repository = Repository(context)
61+
62+
# Add a user
63+
user = User(name="John Doe")
64+
context.add(user)
65+
await context.commit()
66+
67+
# Query users
68+
users = await repository.find(GetAllUsers())
69+
print(f"Found {len(users)} users")
70+
71+
asyncio.run(main())
3372
```
3473

3574
## Installation
@@ -73,22 +112,87 @@ nox -s datajam
73112

74113
The library follows the same architectural patterns as the .NET DataJam:
75114

76-
- **Core abstractions** (`datajam` package): Interfaces and base classes
77-
- **SQLAlchemy implementation** (`datajam_sqlalchemy` package): SQLAlchemy-specific implementations
78-
- **Testing utilities** (`datajam_testing` package): Test helpers and patterns
115+
### Core Abstractions (`datajam` package)
116+
- `IDataContext` - Primary abstraction combining data source, unit of work, and context management
117+
- `IRepository` - Repository pattern for command and query execution
118+
- `IUnitOfWork` - Transaction boundary management
119+
- `IQuery[T]` / `IScalar[T]` - Query pattern interfaces
120+
- `ICommand` - Command pattern for operations without return values
121+
- `IDomain` - Domain configuration and mapping
122+
123+
### SQLAlchemy Implementation (`datajam_sqlalchemy` package)
124+
- `DataContext` - SQLAlchemy-based implementation with async session management
125+
- `Domain` - SQLAlchemy domain configuration
126+
- `Repository` - Repository implementation
127+
- `QueryableImpl` - Fluent query API implementation
128+
- Oracle-specific utilities for case sensitivity and sequences
129+
130+
### Testing Infrastructure
131+
- TestContainers integration for database testing
132+
- Family domain test entities (Person, Father, Mother, Child)
133+
- Pytest fixtures and async test support
134+
135+
## Oracle Database Support
136+
137+
DataJam Python provides first-class Oracle support with:
138+
139+
```python
140+
from datajam_sqlalchemy import create_oracle_connection_string, OracleNamingConvention
141+
142+
# Oracle connection
143+
connection_string = create_oracle_connection_string(
144+
host="localhost",
145+
port=1521,
146+
service_name="XE",
147+
username="system",
148+
password="oracle"
149+
)
150+
151+
# Oracle naming conventions (automatic uppercase conversion)
152+
class OracleDomain(Domain):
153+
def __init__(self, engine):
154+
super().__init__(
155+
connection_string=str(engine.url),
156+
engine=engine,
157+
mapping_configurator=OracleMappingConfigurator()
158+
)
159+
160+
class OracleMappingConfigurator(IConfigureDomainMappings[MetaData]):
161+
def configure(self, metadata):
162+
OracleNamingConvention.configure_metadata_for_oracle(metadata)
163+
```
79164

80165
## Testing
81166

82-
Integration tests use TestContainers for database provisioning, ensuring tests run against real database instances:
167+
Integration tests use TestContainers for database provisioning:
83168

84169
```bash
85170
# Run all tests
86171
nox -s test-all
87172

88173
# Run only Oracle integration tests
89174
nox -s test-integration -- tests/integration/oracle/
175+
176+
# Run with coverage
177+
nox -s test-all -- --cov-report=html
90178
```
91179

180+
### Running Oracle Tests
181+
182+
The Oracle integration tests require Docker and will automatically:
183+
1. Start an Oracle Free container
184+
2. Create test schema and tables
185+
3. Run integration tests
186+
4. Clean up containers
187+
188+
## Contributing
189+
190+
1. Fork the repository
191+
2. Create a feature branch
192+
3. Make your changes
193+
4. Run the full test suite: `nox -s datajam`
194+
5. Submit a pull request
195+
92196
## License
93197

94198
MIT License - see LICENSE file for details.

‎python/tests/integration/oracle/oracle_container_manager.py‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,6 @@
1111
if TYPE_CHECKING:
1212
from collections.abc import AsyncGenerator
1313

14-
from datajam_sqlalchemy import create_oracle_connection_string
15-
16-
from .container_constants import (
17-
ORACLE_IMAGE,
18-
ORACLE_PASSWORD,
19-
ORACLE_PORT,
20-
ORACLE_SERVICE_NAME,
21-
ORACLE_USERNAME,
22-
)
2314

2415
logger = logging.getLogger(__name__)
2516

@@ -42,9 +33,7 @@ async def start_container(self) -> None:
4233
logger.info("Starting Oracle container...")
4334

4435
# Create and start the Oracle container using the free image
45-
self._container = OracleDbContainer(
46-
image="gvenzl/oracle-free:slim" # Use the official free image
47-
)
36+
self._container = OracleDbContainer(image="gvenzl/oracle-free:slim") # Use the official free image
4837

4938
# Start the container (this will block until ready)
5039
self._container.start()
@@ -78,6 +67,7 @@ async def _test_connection(self) -> None:
7867

7968
try:
8069
from sqlalchemy import text
70+
8171
async with self._engine.begin() as conn:
8272
result = await conn.execute(text("SELECT 1 FROM DUAL"))
8373
row = result.fetchone()

0 commit comments

Comments
 (0)