Add flexible group parameter with stem, sibling, and callable support (closes #39, #51, #41)

Fix group_by_prefix collision bug where startswith matching caused
image_130 to match image_1300. Replace with Path.stem-based grouping
in a new splitfolders/grouping.py module.

Add group= parameter to ratio(), fixed(), and kfold() supporting:
- group="stem": auto-discovers group size by filename stem
- group="sibling": splits parallel directories in lockstep
- group=callable: custom grouping function (enables manifest splits)

Add --group CLI argument, mutually exclusive with --group_prefix.
Update README with grouping documentation and examples.

Author: jfilter

README.md

@@ -50,8 +50,10 @@ This should get you started to do some serious deep learning on your data. [Read
 -   The files get shuffled.
 -   A [seed](https://docs.python.org/3/library/random.html#random.seed) makes splits reproducible.
 -   Allows randomized [oversampling](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis) for imbalanced datasets.
--   Optionally group files by prefix.
+-   Optionally group files by prefix or by stem.
 -   Optionally split files by file format(s).
+-   Split parallel directories (e.g. `images/` + `annotations/`) in lockstep.
+-   Custom grouping via callable.
 -   (Should) work on all operating systems.
 
 ## Install
@@ -84,13 +86,15 @@ import splitfolders
 # Split with a ratio.
 # To only split into training and validation set, set a tuple to `ratio`, i.e, `(.8, .2)`.
 splitfolders.ratio("input_folder", output="output",
-    seed=1337, ratio=(.8, .1, .1), group_prefix=None, formats=None, move=False) # default values
+    seed=1337, ratio=(.8, .1, .1), group_prefix=None, group=None,
+    formats=None, move=False) # default values
 
 # Split val/test with a fixed number of items, e.g. `(100, 100)`, for each set.
 # To only split into training and validation set, use a single number to `fixed`, i.e., `10`.
 # Set 3 values, e.g. `(300, 100, 100)`, to limit the number of training values.
 splitfolders.fixed("input_folder", output="output",
-    seed=1337, fixed=(100, 100), oversample=False, group_prefix=None, formats=None, move=False) # default values
+    seed=1337, fixed=(100, 100), oversample=False, group_prefix=None, group=None,
+    formats=None, move=False) # default values
 
 # Use `fixed="auto"` with oversampling to auto-compute the val size from the smallest class.
 # Allocates ~20% of the smallest class to validation, rest to training.
@@ -101,15 +105,106 @@ splitfolders.fixed("input_folder", output="output",
 # Each fold directory contains train/ and val/ subdirectories.
 # Uses symlinks by default to avoid k× disk usage.
 splitfolders.kfold("input_folder", output="output",
-    seed=1337, k=5, group_prefix=None, formats=None, move="symlink") # default values
+    seed=1337, k=5, group_prefix=None, group=None,
+    formats=None, move="symlink") # default values
 ```
 
-Occasionally, you may have things that comprise more than a single file (e.g. picture (.png) + annotation (.txt)).
-`splitfolders` lets you split files into equally-sized groups based on their prefix.
-Set `group_prefix` to the length of the group (e.g. `2`).
-But now _all_ files should be part of groups.
+### Grouping files
 
-Also, there might be some instances when you have multiple file formats in these folders. Provide one or multiple extension(s) to `formats` for splitting the files in a list (e.g. `formats = ['.jpeg','.png']`).
+When your data has multiple files per sample (e.g. an image and its annotation), you need to keep them together during the split. There are several ways to do this.
+
+#### Group by prefix (`group_prefix`)
+
+The legacy approach. Set `group_prefix` to the number of files per group (e.g. `2` for image + annotation pairs). Files are grouped by their filename stem (the part before the extension). All stems must have exactly `group_prefix` files.
+
+```
+input/cats/
+    img1.jpg   img1.txt
+    img2.jpg   img2.txt
+```
+
+```python
+splitfolders.ratio("input", output="output", group_prefix=2)
+```
+
+#### Group by stem (`group="stem"`)
+
+A simpler alternative to `group_prefix`. Automatically groups files that share the same stem and discovers the group size. No need to specify how many files per group — it just requires every stem to have the same count.
+
+```python
+splitfolders.ratio("input", output="output", group="stem")
+```
+
+If every stem has only one file (e.g. a folder of just `.jpg` files), `group="stem"` behaves identically to no grouping.
+
+#### Group by sibling directories (`group="sibling"`)
+
+For datasets where file types live in **parallel directories** rather than alongside each other:
+
+```
+data/
+    images/
+        im_1.jpg
+        im_2.jpg
+    annotations/
+        im_1.xml
+        im_2.xml
+```
+
+Use `group="sibling"` to split all directories in lockstep, matching files across directories by stem:
+
+```python
+splitfolders.ratio("data", output="output", group="sibling")
+```
+
+This produces:
+
+```
+output/
+    train/
+        images/im_1.jpg
+        annotations/im_1.xml
+    val/
+        images/im_2.jpg
+        annotations/im_2.xml
+```
+
+Requirements:
+- The input must have at least 2 subdirectories.
+- Every stem must exist in every subdirectory.
+- Cannot be combined with `oversample=True`.
+
+#### Custom grouping (`group=callable`)
+
+For advanced use cases, pass any callable that takes a list of `Path` objects and returns a list of tuples:
+
+```python
+def my_grouping(files):
+    # Custom logic to group files
+    # Return: list of tuples of Path objects
+    ...
+
+splitfolders.ratio("input", output="output", group=my_grouping)
+```
+
+This also covers **manifest-based splitting** (#41). For example, if you have a CSV that defines train/test assignments:
+
+```python
+def group_from_manifest(files):
+    manifest = load_my_csv("split_manifest.csv")
+    # return list of tuples grouped according to manifest
+    ...
+
+splitfolders.ratio("input", output="output", group=group_from_manifest)
+```
+
+> **Note:** `group_prefix` and `group` are mutually exclusive — setting both raises a `ValueError`.
+
+### File formats
+
+There might be some instances when you have multiple file formats in these folders. Provide one or multiple extension(s) to `formats` for splitting only certain files (e.g. `formats=['.jpeg', '.png']`).
+
+### Move options
 
 Set
 - `move=True` or `move='move'` if you want to move the files instead of copying.
@@ -120,7 +215,7 @@ Set
 
 ```
 Usage:
-    splitfolders [--output] [--ratio] [--fixed] [--kfold] [--seed] [--oversample] [--group_prefix] [--formats] [--move] folder_with_images
+    splitfolders [--output] [--ratio] [--fixed] [--kfold] [--seed] [--oversample] [--group_prefix] [--group] [--formats] [--move] folder_with_images
 Options:
     --output        path to the output folder. defaults to `output`. Get created if non-existent.
     --ratio         the ratio to split. e.g. for train/val/test `.8 .1 .1 --` or for train/val `.8 .2 --`.
@@ -132,12 +227,15 @@ Options:
     --seed          set seed value for shuffling the items. defaults to 1337.
     --oversample    enable oversampling of imbalanced datasets, works only with --fixed.
     --group_prefix  split files into equally-sized groups based on their prefix
+    --group         grouping strategy: 'stem' or 'sibling' (mutually exclusive with --group_prefix)
     --formats       split the files based on specified extension(s)
     --move          move the files instead of copying
     --symlink       symlink(create shortcut) the files instead of copying
 Example:
     splitfolders --ratio .8 .1 .1 -- folder_with_images
     splitfolders --kfold 5 folder_with_images
+    splitfolders --group stem --ratio .8 .1 .1 -- folder_with_images
+    splitfolders --group sibling --ratio .8 .1 .1 -- data_with_parallel_dirs
 ```
 
 Because of some [Python quirks](https://github.com/jfilter/split-folders/issues/19) you have to prepend ` --` after using `--ratio`.

splitfolders/cli.py

@@ -56,13 +56,19 @@ def run():
         default=None,
         help="split files into equally-sized groups based on their prefix",
     )
-    group = parser.add_mutually_exclusive_group()
-    group.add_argument(
+    parser.add_argument(
+        "--group",
+        type=str,
+        default=None,
+        help="grouping strategy: 'stem' or 'sibling'",
+    )
+    move_group = parser.add_mutually_exclusive_group()
+    move_group.add_argument(
         "--move",
         action="store_true",
         help="move the files instead of copying",
     )
-    group.add_argument(
+    move_group.add_argument(
         "--symlink",
         action="store_true",
         help="symlink(create shortcut) the files instead of copying",
@@ -85,11 +91,14 @@ def run():
 
     args = parser.parse_args()
 
+    if args.group_prefix is not None and args.group is not None:
+        parser.error("--group_prefix and --group are mutually exclusive.")
+
     if args.symlink:
         args.move = "symlink"
 
     if args.ratio:
-        ratio(args.input, args.output, args.seed, args.ratio, args.group_prefix, args.move, args.formats)
+        ratio(args.input, args.output, args.seed, args.ratio, args.group_prefix, args.group, args.move, args.formats)
     elif args.fixed:
         if args.fixed == ["auto"]:
             fixed_value = "auto"
@@ -102,6 +111,7 @@ def run():
             fixed_value,
             args.oversample,
             args.group_prefix,
+            args.group,
             args.move,
             args.formats,
         )
@@ -112,6 +122,7 @@ def run():
             args.seed,
             args.kfold,
             args.group_prefix,
+            args.group,
             args.move if args.move else "symlink",
             args.formats,
         )

splitfolders/grouping.py

@@ -0,0 +1,101 @@
+from collections import defaultdict
+
+
+def group_by_prefix(files, len_pairs):
+    """Groups files by Path.stem, validates each group has len_pairs members."""
+    stem_groups = defaultdict(list)
+    for f in files:
+        stem_groups[f.stem].append(f)
+    for stem, group in stem_groups.items():
+        if len(group) != len_pairs:
+            raise ValueError(
+                f"Expected {len_pairs} files with stem '{stem}', found {len(group)}"
+            )
+    return [tuple(sorted(g)) for g in sorted(stem_groups.values(), key=lambda g: g[0])]
+
+
+def resolve_grouping(files, group_prefix=None, group=None):
+    """Central dispatcher. Validates mutual exclusivity, routes to the right strategy."""
+    if group_prefix is not None and group is not None:
+        raise ValueError("Cannot use both `group_prefix` and `group`.")
+    if group_prefix is not None:
+        return group_by_prefix(files, group_prefix)
+    if group is None:
+        return files
+    if group == "stem":
+        return group_by_stem(files)
+    if callable(group):
+        return group(files)
+    if group == "sibling":
+        return files  # sibling handled at orchestration level, not here
+    raise ValueError(f"Unknown group value: {group!r}.")
+
+
+def group_by_stem(files):
+    """Groups files by stem. Auto-discovers group size. Validates all groups same size.
+    If group size is 1 (no actual grouping), returns flat list of Paths."""
+    stem_groups = defaultdict(list)
+    for f in files:
+        stem_groups[f.stem].append(f)
+
+    sizes = {len(g) for g in stem_groups.values()}
+    if len(sizes) != 1:
+        size_details = {stem: len(g) for stem, g in stem_groups.items()}
+        raise ValueError(
+            f"All stems must have the same number of files. Found sizes: {size_details}"
+        )
+
+    group_size = sizes.pop()
+    if group_size == 1:
+        return sorted(files)
+
+    return [tuple(sorted(g)) for g in sorted(stem_groups.values(), key=lambda g: g[0])]
+
+
+def setup_sibling_files(input_dir, seed, formats=None):
+    """Lists type dirs, groups files by stem across all dirs.
+    Validates every stem exists in every dir. Returns (type_dir_names, groups)."""
+    from .utils import list_dirs, list_files
+
+    type_dirs = sorted(list_dirs(input_dir))
+    if len(type_dirs) < 2:
+        raise ValueError(
+            f"group='sibling' requires at least 2 subdirectories, found {len(type_dirs)}."
+        )
+
+    type_dir_names = [d.name for d in type_dirs]
+
+    # Collect stems per type dir
+    stems_per_dir = {}
+    files_per_dir = {}
+    for td in type_dirs:
+        dir_files = list_files(td, formats)
+        files_per_dir[td.name] = {f.stem: f for f in dir_files}
+        stems_per_dir[td.name] = set(files_per_dir[td.name].keys())
+
+    # Validate: every stem must exist in every dir
+    all_stems = set()
+    for s in stems_per_dir.values():
+        all_stems |= s
+
+    for stem in sorted(all_stems):
+        for dir_name, stems in stems_per_dir.items():
+            if stem not in stems:
+                raise ValueError(
+                    f"Stem '{stem}' missing in directory '{dir_name}'. "
+                    "All stems must exist in every subdirectory for group='sibling'."
+                )
+
+    # Build groups: each group is a dict mapping type_dir_name -> Path
+    import random
+
+    random.seed(seed)
+    sorted_stems = sorted(all_stems)
+    random.shuffle(sorted_stems)
+
+    groups = []
+    for stem in sorted_stems:
+        group = tuple(files_per_dir[dn][stem] for dn in type_dir_names)
+        groups.append(group)
+
+    return type_dir_names, groups