Skip to content

Commit d775019

Browse files
committed
Promote cloud/SFTP backends and ship PySide6 GUI
- Move boto3, azure-storage-blob, dropbox, paramiko, and PySide6 out of optional extras and into the required runtime dependencies - Auto-register every backend in build_default_registry, so FA_s3_*, FA_azure_blob_*, FA_dropbox_*, and FA_sftp_* work without opt-in - Add automation_file.ui: PySide6 MainWindow with nine tabs (Local, HTTP, Drive, S3, Azure, Dropbox, SFTP, JSON actions, Servers), a persistent log panel, and QThreadPool-based ActionWorker for background dispatch - Expose launch_ui via lazy facade __getattr__ and wire the `ui` subcommand on `python -m automation_file`; add main_ui.py for quick dev launches - Update README, CLAUDE.md, Sphinx usage/architecture/api docs to reflect non-optional backends and the new GUI; add api/ui.rst - Replace test_optional_backends with test_backends (asserts backends are present in the default registry) and add test_ui_smoke covering launcher, MainWindow, and every tab under the offscreen Qt platform - Clean up unused # type: ignore comments on now-required SDKs and adjust ruff/mypy configs accordingly
1 parent e52a8e7 commit d775019

File tree

104 files changed

+2224
-324
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

104 files changed

+2224
-324
lines changed

CLAUDE.md

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -36,24 +36,32 @@ automation_file/
3636
│ │ ├── search_ops.py
3737
│ │ ├── share_ops.py
3838
│ │ └── upload_ops.py
39-
│ ├── s3/ # Optional — pip install automation_file[s3]
40-
│ │ ├── client.py # S3Client (lazy boto3 import)
39+
│ ├── s3/ # S3 (boto3) — auto-registered in build_default_registry()
40+
│ │ ├── client.py # S3Client
4141
│ │ ├── upload_ops.py
4242
│ │ ├── download_ops.py
4343
│ │ ├── delete_ops.py
4444
│ │ └── list_ops.py
45-
│ ├── azure_blob/ # Optional — pip install automation_file[azure]
45+
│ ├── azure_blob/ # Azure Blob — auto-registered in build_default_registry()
4646
│ │ └── {client,upload,download,delete,list}_ops.py
47-
│ ├── dropbox_api/ # Optionalpip install automation_file[dropbox]
47+
│ ├── dropbox_api/ # Dropboxauto-registered in build_default_registry()
4848
│ │ └── {client,upload,download,delete,list}_ops.py
49-
│ └── sftp/ # Optional — pip install automation_file[sftp]
49+
│ └── sftp/ # SFTP (paramiko + RejectPolicy) — auto-registered in build_default_registry()
5050
│ └── {client,upload,download,delete,list}_ops.py
5151
├── server/
5252
│ ├── tcp_server.py # Loopback-only TCP server executing JSON actions (optional shared-secret auth)
5353
│ └── http_server.py # Loopback-only HTTP server (POST /actions, optional Bearer auth)
5454
├── project/
5555
│ ├── project_builder.py # ProjectBuilder (Builder pattern)
5656
│ └── templates.py # Scaffolding templates
57+
├── ui/ # PySide6 GUI (required dep)
58+
│ ├── launcher.py # launch_ui(argv) — boots QApplication + MainWindow
59+
│ ├── main_window.py # MainWindow — tabbed control surface over every feature
60+
│ ├── worker.py # ActionWorker(QRunnable) + _WorkerSignals
61+
│ ├── log_widget.py # LogPanel — timestamped, read-only log stream
62+
│ └── tabs/ # One tab per domain: local / http / drive / s3 /
63+
│ # azure / dropbox / sftp /
64+
│ # JSON actions / servers
5765
└── utils/
5866
└── file_discovery.py # Recursive file listing by extension
5967
```
@@ -73,7 +81,9 @@ automation_file/
7381
- `CallbackExecutor` — runs a registered trigger, then a user callback, sharing the executor's registry.
7482
- `PackageLoader` — imports a package by name and registers its top-level functions / classes / builtins as `<package>_<member>`.
7583
- `GoogleDriveClient` — wraps OAuth2 credential loading; exposes `service` lazily. `later_init(token_path, credentials_path)` bootstraps; `require_service()` raises if not initialised.
76-
- `S3Client` / `AzureBlobClient` / `DropboxClient` / `SFTPClient` — lazy-import singleton wrappers around the optional SDKs. Each exposes `later_init(...)` plus `close()` where relevant. Operations are registered via `register_<backend>_ops(registry)`.
84+
- `S3Client` / `AzureBlobClient` / `DropboxClient` / `SFTPClient` — singleton wrappers around the required SDKs. Each exposes `later_init(...)` plus `close()` where relevant. Their ops are auto-registered by `build_default_registry()`; `register_<backend>_ops(registry)` is still exported so callers can populate custom registries.
85+
- `MainWindow` — PySide6 tabbed control surface (`ui/main_window.py`). Nine tabs — Local, HTTP, Google Drive, S3, Azure Blob, Dropbox, SFTP, JSON actions, Servers — share a `LogPanel` and dispatch work through `ActionWorker(QRunnable)` on the global `QThreadPool`.
86+
- `launch_ui(argv=None)` — boots / reuses a `QApplication`, shows `MainWindow`, and returns the exec code. Exposed lazily on the facade via `__getattr__` so the Qt runtime isn't paid for by non-UI importers.
7787
- `TCPActionServer` — threaded TCP server that deserialises a JSON action list per connection. Defaults to loopback; optional `shared_secret` enforces `AUTH <secret>\n` prefix.
7888
- `HTTPActionServer``ThreadingHTTPServer` exposing `POST /actions`. Defaults to loopback; optional `shared_secret` enforces `Authorization: Bearer <secret>`.
7989
- `Quota` — frozen dataclass capping bytes and wall-clock seconds per action or block (`check_size`, `time_budget` context manager, `wraps` decorator). `0` disables each cap.
@@ -84,7 +94,7 @@ automation_file/
8494

8595
- `main` branch: stable releases, publishes `automation_file` to PyPI (version in `stable.toml`).
8696
- `dev` branch: development, publishes `automation_file_dev` to PyPI (version in `dev.toml`).
87-
- Keep both TOMLs in sync when bumping. `[project.optional-dependencies]` (s3/azure/dropbox/sftp/dev) must also stay in sync.
97+
- Keep both TOMLs in sync when bumping. `dependencies` and `[project.optional-dependencies]` (`dev`) must also stay in sync. Backends (`boto3`, `azure-storage-blob`, `dropbox`, `paramiko`) and `PySide6` are first-class runtime deps — do not move them back under extras.
8898
- CI: GitHub Actions (Windows, Python 3.10 / 3.11 / 3.12) — one matrix workflow per branch: `.github/workflows/ci-dev.yml`, `.github/workflows/ci-stable.yml`.
8999
- CI steps: `lint` (ruff check + ruff format --check + mypy) → `pytest` with coverage → uploads `coverage.xml` as an artifact.
90100
- Stable branch additionally runs a `publish` job on push to `main`: builds the sdist + wheel, `twine check`, `twine upload` using `PYPI_API_TOKEN`, then `gh release create v<version> --generate-notes`.

README.md

Lines changed: 43 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3,16 +3,18 @@
33
A modular automation framework for local file / directory / ZIP operations,
44
SSRF-validated HTTP downloads, remote storage (Google Drive, S3, Azure Blob,
55
Dropbox, SFTP), and JSON-driven action execution over embedded TCP / HTTP
6-
servers. All public functionality is re-exported from the top-level
7-
`automation_file` facade.
6+
servers. Ships with a PySide6 GUI that exposes every feature through tabs.
7+
All public functionality is re-exported from the top-level `automation_file`
8+
facade.
89

910
- Local file / directory / ZIP operations with path traversal guard (`safe_join`)
1011
- Validated HTTP downloads with SSRF protections, retry, and size / time caps
1112
- Google Drive CRUD (upload, download, search, delete, share, folders)
12-
- Optional S3, Azure Blob, Dropbox, and SFTP backends behind extras
13+
- First-class S3, Azure Blob, Dropbox, and SFTP backends — installed by default
1314
- JSON action lists executed by a shared `ActionExecutor` — validate, dry-run, parallel
1415
- Loopback-first TCP **and** HTTP servers that accept JSON command batches with optional shared-secret auth
1516
- Reliability primitives: `retry_on_transient` decorator, `Quota` size / time budgets
17+
- PySide6 GUI (`python -m automation_file ui`) with a tab per backend plus a JSON-action runner
1618
- Rich CLI with one-shot subcommands plus legacy JSON-batch flags
1719
- Project scaffolding (`ProjectBuilder`) for executor-based automations
1820

@@ -47,24 +49,32 @@ flowchart LR
4749
UrlVal[url_validator]
4850
Http[http_download]
4951
Drive["google_drive<br/>client + *_ops"]
50-
S3["s3<br/>(optional)"]
51-
Azure["azure_blob<br/>(optional)"]
52-
Dropbox["dropbox_api<br/>(optional)"]
53-
SFTP["sftp<br/>(optional)"]
52+
S3["s3"]
53+
Azure["azure_blob"]
54+
Dropbox["dropbox_api"]
55+
SFTP["sftp"]
5456
end
5557
5658
subgraph Server["server"]
5759
TCP[TCPActionServer]
5860
HTTP[HTTPActionServer]
5961
end
6062
63+
subgraph UI["ui (PySide6)"]
64+
Launcher[launch_ui]
65+
MainWindow["MainWindow<br/>9-tab control surface"]
66+
end
67+
6168
subgraph Project["project / utils"]
6269
Builder[ProjectBuilder]
6370
Templates[templates]
6471
Discovery[file_discovery]
6572
end
6673
6774
User --> Public
75+
User --> Launcher
76+
Launcher --> MainWindow
77+
MainWindow --> Public
6878
Public --> Executor
6979
Public --> Callback
7080
Public --> Loader
@@ -109,20 +119,18 @@ through the same shared registry instance exposed as `executor.registry`.
109119
pip install automation_file
110120
```
111121

112-
Optional cloud backends (lazy-imported — install only what you need):
122+
A single install pulls in every backend (Google Drive, S3, Azure Blob, Dropbox,
123+
SFTP) and the PySide6 GUI — no extras required for day-to-day use.
113124

114125
```bash
115-
pip install "automation_file[s3]" # boto3
116-
pip install "automation_file[azure]" # azure-storage-blob
117-
pip install "automation_file[dropbox]" # dropbox
118-
pip install "automation_file[sftp]" # paramiko
119-
pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov
126+
pip install "automation_file[dev]" # ruff, mypy, pre-commit, pytest-cov, build, twine
120127
```
121128

122129
Requirements:
123130
- Python 3.10+
124-
- `google-api-python-client`, `google-auth-oauthlib` (for Drive)
125-
- `requests`, `tqdm` (for HTTP download with progress)
131+
- Bundled dependencies: `google-api-python-client`, `google-auth-oauthlib`,
132+
`requests`, `tqdm`, `boto3`, `azure-storage-blob`, `dropbox`, `paramiko`,
133+
`PySide6`
126134

127135
## Usage
128136

@@ -213,12 +221,14 @@ target = safe_join("/data/jobs", user_supplied_path)
213221
# raises PathTraversalException if the resolved path escapes /data/jobs.
214222
```
215223

216-
### Optional cloud backends
224+
### Cloud / SFTP backends
225+
Every backend is auto-registered by `build_default_registry()`, so `FA_s3_*`,
226+
`FA_azure_blob_*`, `FA_dropbox_*`, and `FA_sftp_*` actions are available out
227+
of the box — no separate `register_*_ops` call needed.
228+
217229
```python
218-
from automation_file import executor
219-
from automation_file.remote.s3 import register_s3_ops, s3_instance
230+
from automation_file import execute_action, s3_instance
220231

221-
register_s3_ops(executor.registry)
222232
s3_instance.later_init(region_name="us-east-1")
223233

224234
execute_action([
@@ -230,6 +240,19 @@ All backends (`s3`, `azure_blob`, `dropbox_api`, `sftp`) expose the same five
230240
operations: `upload_file`, `upload_dir`, `download_file`, `delete_*`, `list_*`.
231241
SFTP uses `paramiko.RejectPolicy` — unknown hosts are rejected, not auto-added.
232242

243+
### GUI
244+
```bash
245+
python -m automation_file ui # or: python main_ui.py
246+
```
247+
248+
```python
249+
from automation_file import launch_ui
250+
launch_ui()
251+
```
252+
253+
Tabs: Local, HTTP, Google Drive, S3, Azure Blob, Dropbox, SFTP, JSON actions,
254+
Servers. A persistent log panel at the bottom streams every result and error.
255+
233256
### Scaffold an executor-based project
234257
```python
235258
from automation_file import create_project_dir
@@ -241,6 +264,7 @@ create_project_dir("my_workflow")
241264

242265
```bash
243266
# Subcommands (one-shot operations)
267+
python -m automation_file ui
244268
python -m automation_file zip ./src out.zip --dir
245269
python -m automation_file unzip out.zip ./restored
246270
python -m automation_file download https://example.com/file.bin file.bin

automation_file/__init__.py

Lines changed: 100 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,11 @@
44
singleton is re-exported from here, so callers only ever need
55
``from automation_file import X``.
66
"""
7+
78
from __future__ import annotations
89

10+
from typing import TYPE_CHECKING, Any
11+
912
from automation_file.core.action_executor import (
1013
ActionExecutor,
1114
add_command_to_executor,
@@ -42,6 +45,16 @@
4245
zip_info,
4346
)
4447
from automation_file.project.project_builder import ProjectBuilder, create_project_dir
48+
from automation_file.remote.azure_blob import (
49+
AzureBlobClient,
50+
azure_blob_instance,
51+
register_azure_blob_ops,
52+
)
53+
from automation_file.remote.dropbox_api import (
54+
DropboxClient,
55+
dropbox_instance,
56+
register_dropbox_ops,
57+
)
4558
from automation_file.remote.google_drive.client import GoogleDriveClient, driver_instance
4659
from automation_file.remote.google_drive.delete_ops import drive_delete_file
4760
from automation_file.remote.google_drive.download_ops import (
@@ -66,6 +79,8 @@
6679
drive_upload_to_folder,
6780
)
6881
from automation_file.remote.http_download import download_file
82+
from automation_file.remote.s3 import S3Client, register_s3_ops, s3_instance
83+
from automation_file.remote.sftp import SFTPClient, register_sftp_ops, sftp_instance
6984
from automation_file.remote.url_validator import validate_http_url
7085
from automation_file.server.http_server import HTTPActionServer, start_http_action_server
7186
from automation_file.server.tcp_server import (
@@ -74,37 +89,101 @@
7489
)
7590
from automation_file.utils.file_discovery import get_dir_files_as_list
7691

92+
if TYPE_CHECKING:
93+
from automation_file.ui.launcher import launch_ui as launch_ui
94+
7795
# Shared callback executor + package loader wired to the shared registry.
7896
callback_executor: CallbackExecutor = CallbackExecutor(executor.registry)
7997
package_manager: PackageLoader = PackageLoader(executor.registry)
8098

99+
100+
def __getattr__(name: str) -> Any:
101+
if name == "launch_ui":
102+
from automation_file.ui.launcher import launch_ui as _launch_ui
103+
104+
return _launch_ui
105+
raise AttributeError(f"module 'automation_file' has no attribute {name!r}")
106+
107+
81108
__all__ = [
82109
# Core
83-
"ActionExecutor", "ActionRegistry", "CallbackExecutor", "PackageLoader",
84-
"Quota", "build_default_registry", "execute_action", "execute_action_parallel",
85-
"execute_files", "validate_action", "retry_on_transient",
86-
"add_command_to_executor", "read_action_json", "write_action_json",
87-
"executor", "callback_executor", "package_manager",
110+
"ActionExecutor",
111+
"ActionRegistry",
112+
"CallbackExecutor",
113+
"PackageLoader",
114+
"Quota",
115+
"build_default_registry",
116+
"execute_action",
117+
"execute_action_parallel",
118+
"execute_files",
119+
"validate_action",
120+
"retry_on_transient",
121+
"add_command_to_executor",
122+
"read_action_json",
123+
"write_action_json",
124+
"executor",
125+
"callback_executor",
126+
"package_manager",
88127
# Local
89-
"copy_file", "rename_file", "remove_file", "copy_all_file_to_dir",
90-
"copy_specify_extension_file", "create_file",
91-
"copy_dir", "create_dir", "remove_dir_tree", "rename_dir",
92-
"zip_dir", "zip_file", "zip_info", "zip_file_info", "set_zip_password",
93-
"unzip_file", "read_zip_file", "unzip_all",
94-
"safe_join", "is_within",
128+
"copy_file",
129+
"rename_file",
130+
"remove_file",
131+
"copy_all_file_to_dir",
132+
"copy_specify_extension_file",
133+
"create_file",
134+
"copy_dir",
135+
"create_dir",
136+
"remove_dir_tree",
137+
"rename_dir",
138+
"zip_dir",
139+
"zip_file",
140+
"zip_info",
141+
"zip_file_info",
142+
"set_zip_password",
143+
"unzip_file",
144+
"read_zip_file",
145+
"unzip_all",
146+
"safe_join",
147+
"is_within",
95148
# Remote
96-
"download_file", "validate_http_url",
97-
"GoogleDriveClient", "driver_instance",
98-
"drive_search_all_file", "drive_search_field", "drive_search_file_mimetype",
99-
"drive_upload_dir_to_folder", "drive_upload_to_folder",
100-
"drive_upload_dir_to_drive", "drive_upload_to_drive",
149+
"download_file",
150+
"validate_http_url",
151+
"GoogleDriveClient",
152+
"driver_instance",
153+
"drive_search_all_file",
154+
"drive_search_field",
155+
"drive_search_file_mimetype",
156+
"drive_upload_dir_to_folder",
157+
"drive_upload_to_folder",
158+
"drive_upload_dir_to_drive",
159+
"drive_upload_to_drive",
101160
"drive_add_folder",
102-
"drive_share_file_to_anyone", "drive_share_file_to_domain", "drive_share_file_to_user",
161+
"drive_share_file_to_anyone",
162+
"drive_share_file_to_domain",
163+
"drive_share_file_to_user",
103164
"drive_delete_file",
104-
"drive_download_file", "drive_download_file_from_folder",
165+
"drive_download_file",
166+
"drive_download_file_from_folder",
167+
"S3Client",
168+
"s3_instance",
169+
"register_s3_ops",
170+
"AzureBlobClient",
171+
"azure_blob_instance",
172+
"register_azure_blob_ops",
173+
"DropboxClient",
174+
"dropbox_instance",
175+
"register_dropbox_ops",
176+
"SFTPClient",
177+
"sftp_instance",
178+
"register_sftp_ops",
105179
# Server / Project / Utils
106-
"TCPActionServer", "start_autocontrol_socket_server",
107-
"HTTPActionServer", "start_http_action_server",
108-
"ProjectBuilder", "create_project_dir",
180+
"TCPActionServer",
181+
"start_autocontrol_socket_server",
182+
"HTTPActionServer",
183+
"start_http_action_server",
184+
"ProjectBuilder",
185+
"create_project_dir",
109186
"get_dir_files_as_list",
187+
# UI (lazy-loaded)
188+
"launch_ui",
110189
]

0 commit comments

Comments
 (0)