Skip to content

Added disordered trace grain data to grain_statistics.csv #1376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tobyallwood wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Added disordered trace grain data to grain_statistics.csv #1376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2eb40c0
Added disordered trace grain data to grain_statistics.csv
tobyallwood Jun 24, 2026
1edfbc6
Updated snapshot
tobyallwood Jul 9, 2026
a2af26c
Updated csv names in docs
tobyallwood Jul 9, 2026
00480bc
linting error fixes
tobyallwood Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/advanced/disordered_tracing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ branches.
![object mask to pruned skeleton](../_static/images/disordered_tracing/overview.png)

This module measures the number of junctions and endpoints for each pruned skeleton object and appends these columns to
the `all_statistics.csv`. In addition, the `all_disordered_segment_statistics.csv` file is produced which measures the
the `grain_statistics.csv`. In addition, the `branch_statistics.csv` file is produced which measures the
length, type, connections, and pixel value (typically height); minimum, middle, median, mean and standard deviation for
each skeleton segment between junctions using [Skan](https://skeleton-analysis.org/stable/index.html). The branch types
are given by:
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/grainstats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## At a Glance - Measures Objects

TopoStats automatically tries to measure the grains (objects of interest) found in the grain finding section, in your
AFM images, and outputs them into the `all_statistics.csv` file.
AFM images, and outputs them into the `grain_statistics.csv` file.

The metrics are briefly summarised in the table below:

Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/nodestats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ skeleton, the effectiveness of automating the joining of skeleton junction point
This module identifies crossing regions from nearby skeleton junctions and analyses each branch emanating out from the
crossing to pair them, then determines the overlying and underlying strand using the full-width half-maximum from each
height trace passing through the crossing. It adds the number of identified crossings and the minimum and average pseudo
confidence values to the `all_statistics.csv`.
confidence values to the `grain_statistics.csv`.

Some quick FYI's:

Expand Down
14 changes: 7 additions & 7 deletions docs/advanced/ordered_tracing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ NodeStats.
This module orders the disordered trace pixel-by-pixel (`topostats` method) or segment-by-segment (`nodestats` method),
giving direction to the trace and creating a path to follow. It adds the number of identified molecules (found by
restarting the trace when using the `nodestats` method) and whether the trace contains endpoints and is therefore
circular or not to the `all_statistics.csv`.
circular or not to the `grain_statistics.csv`.

Some quick FYI's:

Expand Down Expand Up @@ -142,12 +142,12 @@ For each grain, the following new columns are added to the `grainstats.csv` file

For each molecule found by the ordering algorithm(s), the following new columns are added to the `molstats.csv` file:

| Column Name | Description | Data Type |
| --------------- | ------------------------------------------------------------------------------------------------------------------- | --------- |
| `circular` | The number of molecules found by following the tracing paths. Note: This will always be 1 for the TopoStats method. | `integer` |
| `topology` | Whether the disordered trace contains an endpoint. | `bool` |
| `topology_flip` | The number of molecules found by following the tracing paths. Note: This will always be 1 for the TopoStats method. | `integer` |
| `processing` | The method used for ordering. | `str` |
| Column Name | Description | Data Type |
| --------------- | ---------------------------------------------------------- | --------- |
| `circular` | Whether the disordered trace contains an endpoint. | `bool` |
| `topology` | Topological classification of the molecule. | `str` |
| `topology_flip` | Reverse of the topological classification of the molecule. | `str` |
| `processing` | The method used for ordering. | `str` |

 

Expand Down
4 changes: 2 additions & 2 deletions docs/advanced/splining.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ ordering worked successfully, and whether the skeleton matches the underlying sa

This smooths the ordered trace by using an average of splines through the ordered coordinates (`spline` method) or using
the mean coordinate of a rolling window (`rolling_window` method), helping to resolve length errors in jagged in the
skeletons. It adds the contour length and end-to-end euclidean distance to the `all_mol_statistics.csv` and the sum and
average of these respectively to the `all_statistics.csv`.
skeletons. It adds the contour length and end-to-end euclidean distance to the `molecule_statistics.csv` and the sum and
average of these respectively to the `grain_statistics.csv`.

Some quick FYI's:

Expand Down
171 changes: 111 additions & 60 deletions docs/usage/data_dictionary.md
View file
Open in desktop

Large diffs are not rendered by default.

23 changes: 12 additions & 11 deletions docs/usage/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ along with information about how to give feedback, report bugs and cite the soft
Files Found : 1
Successfully Processed : 1 (100.0%)
Configuration : output/config.yaml
All statistics : output/all_statistics.csv
All statistics : output/grain_statistics.csv
Distribution Plots : output/summary_distributions

Email : topostats@sheffield.ac.uk
Expand Down Expand Up @@ -313,10 +313,11 @@ used your own customised configuration file (specifically if you have modified t
At the top level of the output directory are a few files produced:

- `config.yaml` : a copy of the configuration used to process the images.
- `all_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the grain statistics.
- `all_disordered_segment_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the branched skeleton
statistics.
- `all_mol_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the molecule statistics.
- `image_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the image statistics.
- `grain_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the grain statistics.
- `molecule_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the molecule statistics.
- `branch_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the branch statistics.
- `matched_branch_statistics.csv` : a Comma Separated Variable ASCII plain-text file of the matched branch statistics.

**Note:** - If all grains / branch segments of a column have a `None` or `NaN` value, the column will not be present in
the output `.csv` file.
Expand All @@ -342,11 +343,11 @@ one under `level1/a`...

```bash
[4.0K Nov 15 14:06] output
|-- [ 381 Nov 15 14:06] output/all_statistics.csv
|-- [ 733 Nov 15 14:06] output/all_disordered_tracing_statistics.csv
|-- [ 254 Nov 15 14:06] output/all_mol_statistics.csv
|-- [ 381 Nov 15 14:06] output/grain_statistics.csv
|-- [ 733 Nov 15 14:06] output/branch_statistics.csv
|-- [ 254 Nov 15 14:06] output/molecule_statistics.csv
|-- [7.4K Nov 15 14:06] output/config.yaml
|-- [ 222 Nov 15 14:06] output/image_stats.csv
|-- [ 222 Nov 15 14:06] output/image_statistics.csv
|-- [4.0K Nov 15 14:06] output/level1
| |-- [4.0K Nov 15 14:06] output/level1/a
| | |-- [4.0K Nov 15 14:06] output/level1/a/Processed
Expand Down Expand Up @@ -375,13 +376,13 @@ is `output/summary_distributions`. If you have used a custom configuration file
`summary_distributions` nested under the directory specified for the `output`, e.g. if you used the current directory as
output you will have a `summary_distributions` directory present.

Sometimes you may have a `all_statistics.csv` from a run and wish to plot distributions of additional statistics that
Sometimes you may have a `grain_statistics.csv` from a run and wish to plot distributions of additional statistics that
were not already plotted. This can be achieved using the command line programme `toposum` which is included.

**NB** Because of the inherent complexity of plots this script is, by design, limited in the scope to which plots can be
configured. It uses the plotting library [Seaborn](https://seaborn.pydata.org/) (which is built on top of
[Matplotlib](https://matplotlib.org/)) to produce basic plots, which are not intended for publication. If you want to
tweak or customise plots it is recommended to load `all_statistics.csv` into a [Jupyter Notebook](https://jupyter.org)
tweak or customise plots it is recommended to load `grain_statistics.csv` into a [Jupyter Notebook](https://jupyter.org)
and generate the plots you want there. A sample notebook is included to show how to do this.

### Configuring Summary Plots
Expand Down
10 changes: 5 additions & 5 deletions tests/__snapshots__/test_processing.ambr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@
# ---
# name: test_process_scan.1
'''
centre_x centre_y radius_min radius_max radius_mean radius_median height_min height_max height_median height_mean volume area area_cartesian_bbox smallest_bounding_width smallest_bounding_length smallest_bounding_area aspect_ratio max_feret min_feret num_crossings mean_crossing_confidence min_crossing_confidence num_mols writhe_string curvature_grain_num_turns curvature_mean curvature_max curvature_min curvature_std curvature_var curvature_total curvature_median curvature_iqr curvature_90th image
grain_number class subgrain
0 1 0 8.7079e-08 3.5579e-08 3.9431e-09 2.5631e-08 1.6016e-08 1.6680e-08 9.1991e-10 2.6422e-09 1.5338e-09 1.5341e-09 1.0543e-24 6.8721e-16 1.3198e-15 2.0539e-08 5.0379e-08 1.0347e-15 4.0769e-01 5.0379e-08 2.0539e-08 1.0000e+00 NaN NaN 2 0 6.4218e-02 2.5648e-01 0.0000e+00 7.0853e-02 5.0200e-03 8.6694e+00 4.6522e-02 2.0314e-01 1.7462e-01 minicircle_small
1 1 0 8.6581e-08 7.2337e-08 6.8951e-09 2.7188e-08 1.6272e-08 1.6263e-08 9.0630e-10 2.4586e-09 1.6144e-09 1.6264e-09 1.0352e-24 6.3645e-16 1.5931e-15 2.0174e-08 5.1212e-08 1.0332e-15 3.9394e-01 5.1262e-08 2.0174e-08 NaN NaN NaN 1 NA 0 1.0999e-01 3.3925e-01 2.3270e-03 7.2083e-02 5.1960e-03 1.2649e+01 1.0152e-01 2.6693e-01 2.1756e-01 minicircle_small
2 1 0 3.8100e-08 7.7556e-08 9.9461e-09 2.3654e-08 1.7561e-08 1.8364e-08 9.0641e-10 2.1066e-09 1.5939e-09 1.5493e-09 1.1192e-24 7.2236e-16 1.5462e-15 3.3592e-08 4.1496e-08 1.3940e-15 8.0952e-01 4.4405e-08 3.2528e-08 NaN NaN NaN 1 NA 1 8.1232e-02 3.4719e-01 5.9000e-05 7.1972e-02 5.1800e-03 1.3891e+01 6.3881e-02 2.3120e-01 1.9778e-01 minicircle_small
centre_x centre_y radius_min radius_max radius_mean radius_median height_min height_max height_median height_mean volume area area_cartesian_bbox smallest_bounding_width smallest_bounding_length smallest_bounding_area aspect_ratio max_feret min_feret num_crossings mean_crossing_confidence min_crossing_confidence num_mols writhe_string curvature_grain_num_turns curvature_mean curvature_max curvature_min curvature_std curvature_var curvature_total curvature_median curvature_iqr curvature_90th grain_endpoints grain_junctions total_branch_length grain_width_mean image
grain_number class subgrain
0 1 0 8.7079e-08 3.5579e-08 3.9431e-09 2.5631e-08 1.6016e-08 1.6680e-08 9.1991e-10 2.6422e-09 1.5338e-09 1.5341e-09 1.0543e-24 6.8721e-16 1.3198e-15 2.0539e-08 5.0379e-08 1.0347e-15 4.0769e-01 5.0379e-08 2.0539e-08 1.0000e+00 NaN NaN 2 0 6.4218e-02 2.5648e-01 0.0000e+00 7.0853e-02 5.0200e-03 8.6694e+00 4.6522e-02 2.0314e-01 1.7462e-01 1 1 8.5729e-08 9.5493e-09 minicircle_small
1 1 0 8.6581e-08 7.2337e-08 6.8951e-09 2.7188e-08 1.6272e-08 1.6263e-08 9.0630e-10 2.4586e-09 1.6144e-09 1.6264e-09 1.0352e-24 6.3645e-16 1.5931e-15 2.0174e-08 5.1212e-08 1.0332e-15 3.9394e-01 5.1262e-08 2.0174e-08 NaN NaN NaN 1 NA 0 1.0999e-01 3.3925e-01 2.3270e-03 7.2083e-02 5.1960e-03 1.2649e+01 1.0152e-01 2.6693e-01 2.1756e-01 0 0 7.3054e-08 7.6878e-09 minicircle_small
2 1 0 3.8100e-08 7.7556e-08 9.9461e-09 2.3654e-08 1.7561e-08 1.8364e-08 9.0641e-10 2.1066e-09 1.5939e-09 1.5493e-09 1.1192e-24 7.2236e-16 1.5462e-15 3.3592e-08 4.1496e-08 1.3940e-15 8.0952e-01 4.4405e-08 3.2528e-08 NaN NaN NaN 1 NA 1 8.1232e-02 3.4719e-01 5.9000e-05 7.1972e-02 5.1800e-03 1.3891e+01 6.3881e-02 2.3120e-01 1.9778e-01 0 0 1.0331e-07 7.8754e-09 minicircle_small
'''
# ---
# name: test_process_scan.2
Expand Down
35 changes: 28 additions & 7 deletions topostats/processing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1011,7 +1011,7 @@ def get_out_paths(
return core_out_path, filter_out_path, grain_out_path, tracing_out_path


def process_scan(
def process_scan( # noqa: C901
topostats_object: TopoStats,
base_dir: str | Path,
filter_config: dict,
Expand Down Expand Up @@ -1171,6 +1171,7 @@ def process_scan(
if topostats_object.grain_crops is not None and len(topostats_object.grain_crops) > 0:
molecule_stats = {}
disordered_tracing_stats = {}
disordered_traces = {}
# Loop over grains pulling out...
#
# - tracing statistics from disordered traces
Expand All @@ -1179,7 +1180,14 @@ def process_scan(
# Saving to a dictionary which we then flatten
for grain_number, grain_crop in topostats_object.grain_crops.items():
if grain_crop.disordered_trace is not None:
disordered_tracing_stats[grain_number] = grain_crop.disordered_trace.stats
dis_trace = grain_crop.disordered_trace
disordered_tracing_stats[grain_number] = dis_trace.stats
disordered_traces[grain_number] = {
"grain_endpoints": dis_trace.grain_endpoints,
"grain_junctions": dis_trace.grain_junctions,
"total_branch_length": dis_trace.total_branch_length,
"grain_width_mean": dis_trace.grain_width_mean,
}
if grain_crop.ordered_trace is not None and grain_crop.ordered_trace.molecule_data is not None:
molecule_stats[grain_number] = grain_crop.ordered_trace.collate_molecule_statistics()
# Molecule Statistics - convert nested dictionary to DataFrame
Expand Down Expand Up @@ -1232,12 +1240,25 @@ def process_scan(
grain_stats = {
grain_number: grain_crop.stats for grain_number, grain_crop in topostats_object.grain_crops.items()
}
# Add top level statistics from a grain's disordered trace to the grain_stats dictionary
combined_grain_stats = {}
for grain_number in grain_stats:
combined_grain_stats[grain_number] = {}
for class_type in grain_stats[grain_number]:
combined_grain_stats[grain_number][class_type] = {}
for subgrain_number in grain_stats[grain_number][class_type]:
combined_grain_stats[grain_number][class_type][subgrain_number] = {
**grain_stats[grain_number][class_type][subgrain_number],
**disordered_traces[grain_number],
}
grain_stats_df = pd.DataFrame.from_dict(
{
(grain_number, class_type, subgrain_number): grain_stats[grain_number][class_type][subgrain_number]
for grain_number, _ in grain_stats.items()
for class_type, _ in grain_stats[grain_number].items()
for subgrain_number, _ in grain_stats[grain_number][class_type].items()
(grain_number, class_type, subgrain_number): combined_grain_stats[grain_number][class_type][
subgrain_number
]
for grain_number, _ in combined_grain_stats.items()
for class_type, _ in combined_grain_stats[grain_number].items()
for subgrain_number, _ in combined_grain_stats[grain_number][class_type].items()
},
orient="index",
)
Expand Down Expand Up @@ -1436,7 +1457,7 @@ def process_grainstats(
Returns
-------
tuple[str | None, TopoStats, pd.DataFrame | None]
A tuple of the image name, the updated TopoStats object, and the grain statistics DataFrame or None.```
A tuple of the image name, the updated TopoStats object, and the grain statistics DataFrame or None.
"""
# Setup configuration, we use that from the topostats_object.config if not explicitly given an option
config = topostats_object.config.copy()
Expand Down
2 changes: 1 addition & 1 deletion topostats/run_modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@ def process(args: argparse.Namespace | None = None) -> None: # noqa: C901

else:
LOGGER.info(
"Writing 'molecule_statistics.csv', 'branch_statistics.csv' and'matched_branch_statistics.csv' skipped"
"Writing 'molecule_statistics.csv', 'branch_statistics.csv' and 'matched_branch_statistics.csv' skipped"
)

# Write config to file
Expand Down
5 changes: 2 additions & 3 deletions topostats/tracing/ordered_tracing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -597,9 +597,9 @@ def identify_writhes(self) -> tuple[str, dict]:
Returns
-------
writhe_string: str
A string of the whole grain writhe sign
A string of the whole grain writhe sign.
node_to_writhe: dict
a dictionary linking each node to its sign.
A dictionary linking each node to its sign.
"""
# compile all vectors for each node and their z_idx
# - want for each node, ordered vectors according to z_idx
Expand Down Expand Up @@ -655,7 +655,6 @@ def run_nodestats_tracing(self) -> dict[str, npt.NDArray]:

Returns
-------
tuple[list, dict, dict]
dict[str, npt.NDArray]
A dictionary of ordered trace images.
"""
Expand Down
Loading