Skip to content

Commit 4ef7217

Browse files
Feature/concrete-data-collector (#161)
* push * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * property to data * indentation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * model reporter if * abstract * flush fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * input types * separate agent and model collectors * introduce more dynamic functionalities * fixing up the pg functions * s3 changes * changing abstract class * init doc * more doc strings * final docstrings * sync * sync * pre commit fixes * precommit * pre commit fixes * pre commit fixes * error fix * test model * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * precommit * test_init * agent level data * private methods * test conditional collect * local tests * base * pg * collected data empty * fulltests * add dependencies * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pycop dependency issue * toml quotes * multi agent * fixes * agent into agent reporter * isin * test agent reporter string issue * multi selection * better name * collect_step * add tests in conditional colect * test pg * pg test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test yml * build yml * build changes * build changes windows and mac * macos changes * nopassword * better error handling * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add schema name * add schema name as public * commit pg test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * persist comment * age exampleset column * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed agent_reporter pg * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * closed currsor * final test * precommit * precommit * doc for datacollector * precommit fix * windows build * revert to the prev commit * added agent reporter for postgress * final pg test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * precommit and build changes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent f928522 commit 4ef7217

5 files changed

Lines changed: 1078 additions & 25 deletions

File tree

.github/workflows/build.yml

Lines changed: 88 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4,39 +4,50 @@ on:
44
push:
55
branches:
66
- main
7-
- 'release**'
7+
- "release**"
88
paths-ignore:
9-
- '**.md'
9+
- "**.md"
1010
pull_request:
1111
paths-ignore:
12-
- '**.md'
12+
- "**.md"
1313
workflow_dispatch:
1414
schedule:
15-
- cron: '0 6 * * 1'
15+
- cron: "0 6 * * 1"
1616

1717
concurrency:
1818
group: "${{ github.workflow }}-${{ github.head_ref || github.run_id }}"
1919
cancel-in-progress: true
2020

2121
jobs:
22-
test_on_matrix:
23-
name: build (${{ matrix.os }}, py${{ matrix.python-version }})
24-
runs-on: ${{ matrix.os }}-latest
22+
test_on_ubuntu:
23+
name: build (ubuntu, py${{ matrix.python-version }})
24+
runs-on: ubuntu-latest
2525
timeout-minutes: 6
2626

2727
strategy:
2828
fail-fast: false
2929
matrix:
30-
os: [windows, ubuntu, macos]
31-
python-version: ['3.13']
32-
include:
33-
- os: ubuntu
34-
python-version: '3.11'
35-
- os: ubuntu
36-
python-version: '3.12'
30+
python-version: ["3.11", "3.12", "3.13"]
3731

3832
env:
3933
MESA_FRAMES_RUNTIME_TYPECHECKING: "true"
34+
POSTGRES_URI: postgresql://user:password@localhost:5432/testdb
35+
SKIP_PG_TESTS: "false"
36+
37+
services:
38+
postgres:
39+
image: postgres:15
40+
ports:
41+
- 5432:5432
42+
env:
43+
POSTGRES_USER: user
44+
POSTGRES_PASSWORD: password
45+
POSTGRES_DB: testdb
46+
options: >-
47+
--health-cmd="pg_isready"
48+
--health-interval=10s
49+
--health-timeout=5s
50+
--health-retries=5
4051
4152
steps:
4253
- uses: actions/checkout@v4
@@ -53,24 +64,80 @@ jobs:
5364

5465
- name: Install mesa-frames + dev dependencies
5566
run: |
56-
# 1. Install the project itself
5767
uv pip install --system .
58-
# 2. Install everything under the "dev" dependency group
5968
uv pip install --group dev --system
6069
6170
- name: Test with pytest
6271
run: pytest --durations=10 --cov=mesa_frames tests/ --cov-report=xml
6372

64-
- if: matrix.os == 'ubuntu'
65-
name: Codecov
73+
- name: Upload coverage to Codecov
6674
uses: codecov/codecov-action@v4
6775
with:
6876
token: ${{ secrets.CODECOV_TOKEN }}
6977

78+
test_on_mac_windows:
79+
name: build (${{ matrix.os }}, py3.13)
80+
runs-on: ${{ matrix.os }}-latest
81+
timeout-minutes: 10
82+
83+
strategy:
84+
matrix:
85+
include:
86+
- os: windows
87+
skip_pg: true
88+
- os: macos
89+
skip_pg: false
90+
91+
env:
92+
MESA_FRAMES_RUNTIME_TYPECHECKING: "true"
93+
POSTGRES_URI: postgresql://user:password@localhost:5432/testdb
94+
SKIP_PG_TESTS: ${{ matrix.skip_pg }}
95+
96+
steps:
97+
- uses: actions/checkout@v4
98+
99+
- name: Set up Python
100+
uses: actions/setup-python@v5
101+
with:
102+
python-version: "3.13"
103+
104+
- name: Install uv via GitHub Action
105+
uses: astral-sh/setup-uv@v6
106+
with:
107+
cache: true
108+
109+
- name: Install mesa-frames + dev dependencies
110+
run: |
111+
uv pip install --system .
112+
uv pip install --group dev --system
113+
114+
- name: Install and Start PostgreSQL (macOS)
115+
if: matrix.os == 'macos'
116+
run: |
117+
brew install postgresql@15
118+
119+
export PATH="/opt/homebrew/opt/postgresql@15/bin:$PATH"
120+
export PGDATA="/opt/homebrew/var/postgresql@15"
121+
122+
# Ensure a clean database directory
123+
rm -rf $PGDATA
124+
mkdir -p $PGDATA
125+
126+
initdb --username=user --auth=trust --encoding=UTF8 $PGDATA
127+
128+
pg_ctl -D $PGDATA -l logfile start
129+
130+
sleep 5
131+
132+
createdb testdb -U user
133+
shell: bash
134+
135+
- name: Test with pytest
136+
run: pytest --durations=10 --cov=mesa_frames tests/ --cov-report=xml
137+
70138
build:
71139
name: build
72140
runs-on: ubuntu-latest
73-
needs: [test_on_matrix]
141+
needs: [test_on_ubuntu, test_on_mac_windows]
74142
steps:
75-
- name: All matrix builds and tests passed
76-
run: echo "All matrix jobs completed successfully."
143+
- run: echo "All matrix jobs completed successfully."

mesa_frames/abstract/datacollector.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ class AbstractDataCollector(ABC):
6464
_agent_reporters: dict[str, str | Callable] | None
6565
_trigger: Callable[..., bool]
6666
_reset_memory = bool
67-
_storage_uri: Literal["memory:", "csv:", "postgresql:"]
67+
_storage: Literal["memory", "csv", "parquet", "S3-csv", "S3-parquet", "postgresql"]
6868
_frames: list[pl.DataFrame]
6969

7070
def __init__(
@@ -74,7 +74,9 @@ def __init__(
7474
agent_reporters: dict[str, str | Callable] | None = None,
7575
trigger: Callable[[Any], bool] | None = None,
7676
reset_memory: bool = True,
77-
storage: Literal["memory:", "csv:", "postgresql:"] = "memory:",
77+
storage: Literal[
78+
"memory", "csv", "parquet", "S3-csv", "S3-parquet", "postgresql"
79+
] = "memory",
7880
):
7981
"""
8082
Initialize a Datacollector.
@@ -91,15 +93,15 @@ def __init__(
9193
A function(model) -> bool that determines whether to collect data.
9294
reset_memory : bool
9395
Whether to reset in-memory data after flushing. Default is True.
94-
storage : Literal["memory:", "csv:", "postgresql:"]
96+
storage : Literal["memory", "csv", "parquet", "S3-csv", "S3-parquet", "postgresql" ]
9597
Storage backend URI (e.g. 'memory:', 'csv:', 'postgresql:').
9698
"""
9799
self._model = model
98100
self._model_reporters = model_reporters or {}
99101
self._agent_reporters = agent_reporters or {}
100102
self._trigger = trigger or (lambda model: False)
101103
self._reset_memory = reset_memory
102-
self._storage_uri = storage or "memory:"
104+
self._storage = storage or "memory"
103105
self._frames = []
104106

105107
def collect(self) -> None:

0 commit comments

Comments
 (0)