Skip to content

python(feat): add data import api to sift_client#515

Merged
wei-qlu merged 52 commits into
mainfrom
python/data-import
Apr 13, 2026
Merged

python(feat): add data import api to sift_client#515
wei-qlu merged 52 commits into
mainfrom
python/data-import

Conversation

@wei-qlu
Copy link
Copy Markdown
Contributor

@wei-qlu wei-qlu commented Mar 26, 2026

What was changed

  • Added data_import resource with support for CSV and Parquet file imports
  • Added low-level gRPC wrapper for DataImportService (DetectConfig, CreateDataImportFromUpload)
  • detect_config auto-detects column structure, time columns, and data types from a file sample
  • import_from_path handles the full flow: detect config, create import, upload file, and poll for completion via the jobs API
  • Parquet supports both flat dataset and single-channel-per-row layouts, with automatic footer extraction
  • Added Pydantic models in sift_types for all CSV and Parquet config types
  • Import results are returned as Job objects, consistent with existing services

Verification

  • Unit tests for config mutability, get_column (helper), time column validation, extension mapping, and run precedence (Integration tests will not be included for now)
  • Full import cycle tested locally for both CSV and Parquet files

@wei-qlu wei-qlu changed the title python(feat): add data import api python(feat): add data import api to sift_client Mar 26, 2026
@wei-qlu wei-qlu force-pushed the python/data-import branch from dcae808 to 265941c Compare March 27, 2026 20:03
@wei-qlu wei-qlu force-pushed the python/data-import branch from ba16e9e to d27b070 Compare April 1, 2026 21:53
@wei-qlu wei-qlu marked this pull request as ready for review April 7, 2026 23:29
@wei-qlu wei-qlu requested a review from alexluck-sift April 10, 2026 20:47
@wei-qlu
Copy link
Copy Markdown
Contributor Author

wei-qlu commented Apr 10, 2026

data_imports.proto was updated to support tdms and some additions to CSV, will be making updates accordingly

@wei-qlu wei-qlu marked this pull request as draft April 10, 2026 22:51
Comment thread python/lib/sift_client/_internal/util/file.py
Comment thread python/lib/sift_client/resources/data_imports.py
Comment thread python/lib/sift_client/resources/data_imports.py Outdated
Comment thread python/lib/sift_client/sift_types/run.py
Comment thread python/lib/sift_client/sift_types/run.py
Comment thread python/lib/sift_client/sift_types/data_import.py Outdated
Comment thread python/lib/sift_client/sift_types/data_import.py Outdated
@wei-qlu wei-qlu self-assigned this Apr 11, 2026
@wei-qlu wei-qlu requested a review from alexluck-sift April 13, 2026 17:20
@wei-qlu wei-qlu marked this pull request as ready for review April 13, 2026 17:20
Copy link
Copy Markdown
Collaborator

@alexluck-sift alexluck-sift left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@wei-qlu wei-qlu merged commit 7fa55ed into main Apr 13, 2026
22 checks passed
@wei-qlu wei-qlu deleted the python/data-import branch April 13, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants