Skip to content

Commit be2b265

Browse files
cdc-ap66cdc-ap66
authored
Merge pull request #49 from CDCgov/ap66-release_2025_12_10_0
First Official Release! 🎉 This release marks a significant milestone for the CFA DataOps project, providing a robust foundation for data cataloging, ETL pipelines, and reporting. It consolidates months of development into a unified, versioned package ready for broader adoption. Key Highlights: Unified Data Access: datacat interface for seamless dataset access. Automated Reporting: reportcat for generating interactive HTML reports from Jupyter notebooks. Robust CLI Tools: Manage catalogs and datasets effortlessly. Flexible Data Loading: Advanced version filtering for Pandas and Polars DataFrames. See the Release Notes for more details.
2 parents eb381ed + 67a1dc9 commit be2b265

4 files changed

Lines changed: 110 additions & 83 deletions

File tree

‎.github/workflows/release.yaml‎

Lines changed: 3 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -17,103 +17,25 @@ jobs:
1717
with:
1818
fetch-depth: 0
1919

20-
- name: Checkout release
21-
run: |
22-
git fetch origin release
23-
git checkout release
24-
25-
- name: Determine Version
26-
id: get-version
27-
env:
28-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
29-
run: |
30-
VERSION="${GITHUB_REF#refs/tags/}"
31-
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
32-
33-
- name: Determine If Tag Is From Release
34-
id: is-release
35-
env:
36-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
37-
run: |
38-
TAG_COMMIT=$(git rev-parse ${{ github.ref }})
39-
BRANCHES=$(git branch -r --contains "$TAG_COMMIT")
40-
41-
if echo "$BRANCHES" | grep -q 'origin/release'; then
42-
echo "should_continue=true" >> "$GITHUB_OUTPUT"
43-
else
44-
echo "Exiting Workflow: Tag is not from release branch"
45-
echo "IF intending to create a release from tag follow steps under 'Creating A Release' in the README file"
46-
echo "should_continue=false" >> "$GITHUB_OUTPUT"
47-
fi
48-
4920
- name: Setup Python
50-
if: steps.is-release.outputs.should_continue == 'true'
5121
uses: actions/setup-python@v5
5222
with:
5323
python-version: '3.10'
5424

55-
- name: Install Poetry
56-
if: steps.is-release.outputs.should_continue == 'true'
25+
- name: Install poetry
5726
run: |
58-
curl -sSL https://install.python-poetry.org | python3 -
59-
poetry --version
27+
pip install poetry>=2.0
6028
6129
- name: Install Dependencies
62-
if: steps.is-release.outputs.should_continue == 'true'
63-
run: |
64-
poetry install --no-root --with dev
65-
66-
- name: Update pyproject.toml Version
67-
if: steps.is-release.outputs.should_continue == 'true'
68-
env:
69-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
70-
run: |
71-
VERSION=${{ steps.get-version.outputs.version }}
72-
73-
sed -i.bak -E "s/^\s*version\s*=\s*\"[^\"]+\"/version = \"${VERSION}\"/" pyproject.toml
74-
75-
- name: Update Change Log
76-
if: steps.is-release.outputs.should_continue == 'true'
77-
env:
78-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
7930
run: |
80-
VERSION=${{ steps.get-version.outputs.version }}
81-
REPO_URL="https://github.com/${{ github.repository }}"
82-
RELEASE_URL="$REPO_URL/releases/tag/$VERSION"
83-
84-
awk -v tag="## [$VERSION]" -v url="$RELEASE_URL" '
85-
!done && /---/ {
86-
print $0
87-
print ""
88-
print tag
89-
print url
90-
done = 1
91-
next
92-
}
93-
{ print $0 }
94-
' changelog.md > temp.md && mv temp.md changelog.md
31+
poetry install --with dev
9532
9633
- name: Build Wheel
97-
if: steps.is-release.outputs.should_continue == 'true'
9834
run: |
9935
poetry build
10036
10137
- name: Create Release
102-
if: steps.is-release.outputs.should_continue == 'true'
10338
env:
10439
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
10540
run: |
10641
gh release create ${{ github.ref }} dist/* --generate-notes --latest
107-
108-
- name: Commit Change Log and pyproject.toml Updates
109-
if: steps.is-release.outputs.should_continue == 'true'
110-
env:
111-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
112-
run: |
113-
VERSION=${{ steps.get-version.outputs.version }}
114-
115-
git config user.name "github-actions"
116-
git config user.email "github-actions@users.noreply.github.com"
117-
git add pyproject.toml changelog.md
118-
git commit -m "Update Version to ${{ steps.get-version.outputs.version }}"
119-
git push origin release

‎changelog.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,21 @@ The versioning pattern is `YYYY.MM.DD.micro(a/b/{none if release})
88

99
---
1010

11-
## [2025.12.10.0a]
11+
## [2025.12.10.0]
12+
13+
### Summary
14+
15+
**First Official Release!** 🎉
16+
17+
This release marks a significant milestone for the CFA DataOps project, providing a robust foundation for data cataloging, ETL pipelines, and reporting. It consolidates months of development into a unified, versioned package ready for broader adoption.
18+
19+
**Key Highlights:**
20+
- **Unified Data Access**: `datacat` interface for seamless dataset access.
21+
- **Automated Reporting**: `reportcat` for generating interactive HTML reports from Jupyter notebooks.
22+
- **Robust CLI Tools**: Manage catalogs and datasets effortlessly.
23+
- **Flexible Data Loading**: Advanced version filtering for Pandas and Polars DataFrames.
24+
25+
See the [Release Notes](docs/release_notes/v2025.12.10.md) for more details.
1226

1327
### Updated
1428

‎docs/release_notes/v2025.12.10.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Release Notes - v2025.12.10
2+
3+
We're thrilled to announce the first official release of the **CFA DataOps** project! This milestone version `2025.12.10.0` brings a comprehensive suite of tools for data cataloging, ETL pipelines, and reporting, designed to streamline data operations within the CFA environment.
4+
5+
## 🚀 Highlights
6+
7+
* **Unified Data Access**: Seamlessly access datasets across multiple catalogs using the `datacat` interface.
8+
* **Automated Reporting**: Generate client-side rendering interactive HTML reports from Jupyter notebooks with `reportcat`.
9+
* **Robust CLI Tools**: Manage catalogs, datasets, stages, and versions directly from the command line.
10+
* **Flexible Data Loading**: Load data into Pandas or Polars DataFrames with advanced version filtering and selection.
11+
* **Azure Blob Storage Integration**: Built-in support for reading/writing raw and transformed data to Azure Blob Storage.
12+
* **Schema Validation**: Ensure data quality with rigorous schema validation for both raw and transformed datasets.
13+
14+
## 🏗️ Architecture
15+
16+
```
17+
┌─────────────────────────────────────────────────────────────────┐
18+
│ CFA Data Science Ecosystem │
19+
├─────────────────────────────────────────────────────────────────┤
20+
│ │
21+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
22+
│ │ Cloud OPS │────>│ Data OPS │────>│ Catalogs │ │
23+
│ │ │ │ │ │ │ │
24+
│ │ • Compute │ │ • Datacat │ │ • Public │ │
25+
│ │ • BLOB │ │ • Reportcat │ │ • Private │ │
26+
│ │ • Key Vault │ │ • Ledger │ │ • Team-spec. │ │
27+
│ │ • │ │ • Cat init │ │ • workflows │ │
28+
│ │ │ │ • │ │ • reports │ │
29+
│ └──────────────┘ └──────────────┘ └──────────────┘ │
30+
│ │ │ │ │
31+
│ └────────────────────┼────────────────────┘ │
32+
│ V │
33+
│ ┌──────────────────┐ │
34+
│ │ Data Scientists │ │
35+
│ │ & Applications │ │
36+
│ └──────────────────┘ │
37+
└─────────────────────────────────────────────────────────────────┘
38+
```
39+
40+
## 📋 Key Features
41+
42+
### Data Management
43+
* **Catalog Creation**: Initialize new dataset catalog libraries with standardized structures using `dataops_catalog_init`.
44+
* **Multi-Catalog Support**: Install and manage multiple catalog libraries in the same Python environment.
45+
* **Configuration-Driven ETL**: Define ETL pipelines using simple TOML configuration files.
46+
47+
### Data Access & Versioning
48+
* **Version Control**: Retrieve specific data versions using timestamp-based versioning.
49+
* **Advanced Filtering**: Use conditional logic (e.g., `>2024.12.01,<2025.08`, `latest`, `~=2024/11`) to select data versions.
50+
* **Local Download**: Download dataset versions to your local filesystem for offline analysis.
51+
52+
### Reporting & Visualization
53+
* **Jupyter Integration**: Author reports as Jupyter notebooks and convert them to interactive HTML.
54+
* **Visualization Utilities**: Includes plotting functions for lines, points, and intervals, plus PDF report generation.
55+
56+
## 🗃️ Existing and Growing Catalogs
57+
58+
* **Public**: [https://github.com/CDCgov/cfa-catalog-pub](https://github.com/CDCgov/cfa-catalog-pub)
59+
* **Private**: [https://github.com/cdcent/cfa-catalog-private](https://github.com/cdcent/cfa-catalog-private)
60+
61+
## 🛠️ Usage Examples
62+
63+
**List Available Datasets:**
64+
```python
65+
from cfa.dataops import datacat
66+
print(datacat.__namespace_list__)
67+
```
68+
69+
**Load a Dataframe with Version Filtering:**
70+
```python
71+
from cfa.dataops import datacat
72+
df = datacat.public.my_dataset.load.get_dataframe(version=">2024.12.01,<2025.08")
73+
```
74+
75+
**Generate a Report:**
76+
```python
77+
from cfa.dataops.reporting import reportcat
78+
reportcat.examples.dataset_report_ipynb.nb_to_html_file('report.html')
79+
```
80+
81+
## 📚 Documentation
82+
83+
For more detailed information, please refer to our comprehensive documentation:
84+
* [Data User Guide](https://cdcgov.github.io/cfa-dataops/data_user_guide/)
85+
* [Data Developer Guide](https://cdcgov.github.io/cfa-dataops/data_developer_guide/)
86+
* [Managing Catalogs](https://cdcgov.github.io/cfa-dataops/managing_catalogs/)
87+
* [Report Generation](https://cdcgov.github.io/cfa-dataops/report_generation/)
88+
* [CLI Tools](https://cdcgov.github.io/cfa-dataops/cli_tools/)
89+
90+
---
91+
*Thank you to all the contributors who made this release possible!*

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "cfa.dataops"
3-
version = "2025.12.10.0a"
3+
version = "2025.12.10.0"
44
description = "Data cataloging, ETL, modeling, verification, and validation for CFA"
55
authors = [
66
{ name = "Phil Rogers", email = "ap66@cdc.gov" },

0 commit comments

Comments
 (0)