Skip to content

Commit d91c004

Browse files
committed
feat: add batch processing support for processing multiple files
Add comprehensive batch processing functionality to process multiple accelerometer files from a folder in a single command. Changes to src/accelerometer/accProcess.py: - Add matchesExtension() to match file extensions with compression support - Add discoverFiles() to discover files by extension with recursive option - Add processSingleFile() extracted from main() for reusability - Add processBatch() to orchestrate batch processing with error handling - Add --fileExtensions argument (required for batch mode) - Add --recursive argument for subdirectory search - Refactor main() to detect and route file vs directory input - Implement try-finally cleanup to ensure intermediate files deleted on error - Catch SystemExit exceptions to prevent device.py sys.exit() from killing batch - Detect duplicate basenames and exit with error to prevent data loss - Exit with non-zero code when no files found or any files fail - Respect --verbose flag for detailed error reporting in batch mode - Remove unused atexit dependency (cleanup now in finally block) Changes to README.md: - Add batch processing usage section with examples - Update output paths to reflect new outputs/{filename}/ structure - Document --fileExtensions and --recursive options - Explain compression format auto-detection Key features: - Serial processing with per-file error isolation - Comprehensive batch summary with success/failure counts - Exit codes: 255 for no files/duplicates, 1 for any failures - Smart extension matching (case-insensitive, compression aware) - Duplicate basename detection prevents output overwrites - Backward compatible: single file mode unchanged
1 parent 32d1958 commit d91c004

2 files changed

Lines changed: 469 additions & 114 deletions

File tree

README.md

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,13 +32,15 @@ The following instructions make use of Anaconda to meet the minimum requirements
3232
You are all set! The next time that you want to use `accelerometer`, open the Anaconda Prompt and activate the environment (step 4). If you see `(accelerometer)` in front of your prompt, you are ready to go!
3333

3434
## Usage
35+
36+
### Processing a single file
3537
To extract summary movement statistics from an Axivity file (.cwa):
3638

3739
```console
3840
$ accProcess data/sample.cwa.gz
3941

40-
<output written to data/sample-outputSummary.json>
41-
<time series output written to data/sample-timeSeries.csv.gz>
42+
<output written to outputs/sample/sample-summary.json>
43+
<time series output written to outputs/sample/sample-timeSeries.csv.gz>
4244
```
4345

4446
Movement statistics will be stored in a JSON file:
@@ -58,6 +60,35 @@ See [Data Dictionary](https://biobankaccanalysis.readthedocs.io/en/latest/datadi
5860

5961
Actigraph and GENEActiv files are also supported, as well as custom CSV files. See [Usage](https://biobankaccanalysis.readthedocs.io/en/latest/usage.html#basic-usage) for more details.
6062

63+
### Batch processing multiple files
64+
To process multiple files in a folder, you must specify which file extensions to process:
65+
66+
```console
67+
$ accProcess data/folder_with_cwa_files/ --fileExtensions cwa
68+
69+
<outputs written to outputs/file1/, outputs/file2/, etc.>
70+
```
71+
72+
You can specify multiple extensions (comma-separated):
73+
74+
```console
75+
$ accProcess data/my_data/ --fileExtensions cwa,bin,csv
76+
```
77+
78+
The tool automatically includes compressed versions (.gz, .zip, .bz2, .xz), so `--fileExtensions cwa` will match both `.cwa` and `.cwa.gz` files.
79+
80+
To search subdirectories recursively:
81+
82+
```console
83+
$ accProcess data/my_data/ --fileExtensions cwa --recursive True
84+
```
85+
86+
The batch processor will:
87+
- Automatically discover matching files
88+
- Process them serially with full error handling
89+
- Continue processing even if individual files fail
90+
- Provide a detailed summary of successful and failed files
91+
6192
To plot the activity profile:
6293
```console
6394
$ accPlot data/sample-timeSeries.csv.gz

0 commit comments

Comments
 (0)