Merge branch 'develop' into nshekhaw/fix_mu_pytorch_fedprox

Author: nisha987

docs/about/features.rst

@@ -109,4 +109,17 @@ In Federated Learning (FL), Secure Aggregation (SecAgg) is a technique that allo
     :hidden:
 
     features_index/secure_aggregation
-    
+
+.. _federated_analytics:
+
+---------------------
+Federated Analytics
+---------------------
+
+Federated Analytics enables the collection and analysis of data insights across decentralized nodes without compromising data privacy. This feature allows organizations to perform analytics on distributed data while ensuring compliance with privacy regulations. For more info see :doc:`features_index/fed_analytics`
+
+.. toctree::
+    :hidden:
+
+    features_index/fed_analytics
+

docs/about/features_index/fed_analytics.rst

@@ -0,0 +1,182 @@
+.. # Copyright (C) 2020-2024 Intel Corporation
+.. # SPDX-License-Identifier: Apache-2.0
+
+Federated Analytics
+=======================================
+
+Introduction to Federated Analytics
+-------------------------------------
+
+Federated Analytics is a privacy-preserving approach to compute statistics or perform data analysis on distributed datasets without aggregating raw data into a centralized location. This method ensures data security while enabling insights to be drawn from decentralized data sources. For instance, one can compute the mean, frequency distributions, or other statistical measures across datasets located on multiple devices. Federated Analytics is particularly valuable in scenarios where data sharing is restricted due to privacy concerns or regulatory constraints.
+
+OpenFL's Support for Federated Analytics
+------------------------------------------
+
+OpenFL, a flexible framework for Federated Learning, extends its capabilities to support Federated Analytics. By leveraging the federation plan and task runner API, OpenFL enables users to perform analytics tasks across collaborators. These tasks are defined in the ``plan.yaml`` file and distributed to collaborators for execution. The results are then aggregated by the aggregator to provide global insights.
+
+
+Example Workspace: Histogram Calculation using sklearn IRIS Dataset
+------------------------------------------------------------------------------
+
+The Federated Analytics workspace for histogram calculation demonstrates how to compute frequency distributions of specific features across distributed datasets. This workspace leverages the OpenFL framework to ensure privacy-preserving analytics while providing global insights into the data.
+
+**Task Configuration:**
+
+The analytics tasks are defined in the `plan.yaml` file. For example:
+
+.. code-block:: yaml
+    :emphasize-lines: 6,41,43,45
+
+    aggregator:
+      defaults: plan/defaults/aggregator.yaml
+      template: openfl.component.Aggregator
+      settings:
+        last_state_path: save/result.json
+        rounds_to_train: 1 # Number of training rounds (set to 1 for Federated Analytics).
+
+    collaborator:
+      defaults: plan/defaults/collaborator.yaml
+      template: openfl.component.Collaborator
+      settings:
+        use_delta_updates: false
+        opt_treatment: RESET
+
+    data_loader:
+      defaults: plan/defaults/data_loader.yaml
+      template: src.dataloader.IRISInMemory
+      settings:
+        collaborator_count: 2
+        data_group_name: iris
+        batch_size: 150
+
+    task_runner:
+      defaults: plan/defaults/task_runner.yaml
+      template: src.taskrunner.IrisHistogram
+
+    network:
+      defaults: plan/defaults/network.yaml
+
+    assigner:
+      template: openfl.component.RandomGroupedAssigner
+      settings:
+        task_groups:
+          - name: analytics
+            percentage: 1.0
+            tasks:
+              - analytics
+
+    tasks:
+      analytics:
+        function: analytics
+        aggregation_type:
+          template: src.aggregatehistogram.AggregateHistogram
+        kwargs:
+          columns: ['sepal length (cm)', 'sepal width (cm)']
+
+**Note:** The `function` and `aggregation_type.template` fields in the configuration can be replaced with custom implementations to suit specific use cases. This flexibility allows users to define their own analytics logic and aggregation methods tailored to their requirements.
+
+**Data Distribution**: The dataset is distributed across collaborators, with each collaborator holding a local shard of the data.
+
+**Local Computation**: Each collaborator computes the histogram for the specified feature(s) on its local data shard. This ensures that raw data never leaves the collaborator's environment.
+
+**Aggregation**: The aggregator collects the histograms from all collaborators and combines them to compute the global histogram. The aggregated results are saved in `save/result.json`. This file provides a global view of the frequency distribution for the selected feature, computed in a privacy-preserving manner.
+
+
+By following this structured approach, the Federated Analytics workspace enables secure and efficient computation of histograms across distributed datasets.
+
+Detailed Instructions
+---------------------
+
+Workspace Setup and Federation Run
+
+Create a workspace for analytics (for example, using the federated_analytics/histogram template):
+
+.. code-block:: bash
+
+    fx workspace create --prefix ./analytics_workspace --template federated_analytics/histogram
+    cd analytics_workspace
+    fx workspace certify
+    fx aggregator generate-cert-request
+    fx aggregator certify --silent
+
+Initialize the plan normally:
+
+.. code-block:: bash
+
+    fx plan initialize
+
+Run the federation using your collaborators. For example:
+
+.. code-block:: bash
+
+    fx collaborator create -n collaborator1 -d 1
+    fx collaborator generate-cert-request -n collaborator1
+    fx collaborator certify -n collaborator1 --silent
+
+    fx collaborator create -n collaborator2 -d 2
+    fx collaborator generate-cert-request -n collaborator2
+    fx collaborator certify -n collaborator2 --silent
+
+    fx aggregator start > ~/fx_aggregator.log 2>&1 &
+    fx collaborator start -n collaborator1 > ~/collab1.log 2>&1 &
+    fx collaborator start -n collaborator2 > ~/collab2.log 2>&1 &
+
+Once the federation run is complete, the results will be saved.
+
+The result file `save/result.json` contains the aggregated histogram data. For example:
+
+.. code-block:: json
+
+    {
+        "sepal length (cm) histogram": [
+            0.0,
+            0.0,
+            9.0,
+            50.0,
+            56.0,
+            28.0,
+            7.0,
+            0.0,
+            0.0
+        ],
+        "sepal length (cm) bins": [
+            4.0,
+            5.777777671813965,
+            7.55555534362793,
+            9.333333015441895,
+            11.11111068725586,
+            12.88888931274414,
+            14.666666984558105,
+            16.44444465637207,
+            18.22222137451172,
+            20.0
+        ],
+        "sepal width (cm) histogram": [
+            47.0,
+            91.0,
+            12.0,
+            0.0,
+            0.0,
+            0.0,
+            0.0,
+            0.0,
+            0.0
+        ],
+        "sepal width (cm) bins": [
+            4.0,
+            5.777777671813965,
+            7.55555534362793,
+            9.333333015441895,
+            11.11111068725586,
+            12.88888931274414,
+            14.666666984558105,
+            16.44444465637207,
+            18.22222137451172,
+            20.0
+        ]
+    }
+
+
+Conclusion
+----------
+Federated Analytics in OpenFL enables privacy-preserving data analysis on distributed datasets. By leveraging the task runner API and predefined analytics tasks, users can seamlessly compute global statistics without compromising data privacy. This feature simplifies the workflow for distributed data analysis and ensures compliance with privacy regulations.

docs/developer_guide/utilities.rst

@@ -6,6 +6,9 @@ The following are utilities available in Open Federated Learning (OpenFL).
 :doc:`utilities/pki`
     Use the Public Key Infrastructure (PKI) solution workflows to certify the nodes in your federation.
 
+:doc:`utilities/verifiable_datasets`
+    Build and verify datasets composed of multiple data sources.
+
 :doc:`utilities/splitters_data`
     Split your data to run your federation from a single dataset.
 
@@ -17,5 +20,7 @@ The following are utilities available in Open Federated Learning (OpenFL).
    :hidden:
 
    utilities/pki
+   utilities/verifiable_datasets
    utilities/splitters_data
-   utilities/timeouts
+   utilities/timeouts
+   

docs/developer_guide/utilities/verifiable_datasets.rst

@@ -0,0 +1,37 @@
+.. # Copyright (C) 2025 Intel Corporation
+.. # SPDX-License-Identifier: Apache-2.0
+
+*************************************
+Verifiable Datasets and Data Sources
+*************************************
+
+.. _verifiable_datasets_overview:
+
+To accommodate for the proliferation of data sources and the need for trusted datasets, OpenFL provides a hierarchy of utility classes to build and verify datasets. 
+This includes an extensible class hierarchy that enables the creation of datasets from various data sources, such as local file system, object storage and others.
+
+The central abstraction is the :code:`VerifiableDatasetInfo` class that encapsulates the dataset's metadata and provides a method for verifying the integrity of the dataset.
+A dataset can be built from multiple data sources (not necessarily of the same type):
+
+.. mermaid:: ../../mermaid/verifiable_dataset_info.mmd
+    :caption: Verifiable Dataset with Multiple Data Sources
+    :align: center
+
+The :code:`VerifiableDatasetInfo` class can then be used to create higher-order dataset classes that enable iterating through multiple data sources, while verifying integrity if required.
+The :code:`root_hash` is used as a reference for integrity when loading items from the the data sources in the :code:`VerifiableDatasetInfo` object.
+
+OpenFL comes with a toolbox of dataset layout classes per ML framework. For PyTorch's :code:`torch.utils.data.Dataset` OpenFL curently provides: 
+
+- :code:`FolderDataset` - represents an iterable folder-layout dataset from a single data source, by implementing the :code:`__getitem__` method.
+- :code:`ImageFolder` - a specialization of the :code:`FolderDataset` that is able to load binary images from a foler-like structure
+- :code:`VerifiableMapStyleDataset` - a base class for map-style datasets that can be built from multiple data sources (as specified by a :code:`VerifiableDatasetInfo` object), including integrity checks.
+- :code:`VerifiableImageFolder` - a specialization of the :code:`VerifiableMapStyleDataset` encapsulating a collection of :code:`ImageFolder` datasets
+
+Note that the all those classes (directly or indirectly) extend :code:`torch.utils.data.DataLoader`, and are therefore compatible with all PyTorch utilities for pre-processing data sets.
+A similar class hierarchy can be created for other ML frameworks that offer dataset utilities, such as TensorFlow.
+
+.. mermaid:: ../../mermaid/verifiable_image_folder.mmd
+    :caption: Dataset hierarchy
+    :align: center
+
+A practical example for the :code:`VerifiableImageFolder` backed by :code:`S3DataSource` is provided in the `s3_histology <https://github.com/securefederatedai/openfl/tree/develop/openfl-workspace/torch/histology_s3>`_ workspace template.

docs/mermaid/verifiable_dataset_info.mmd

@@ -0,0 +1,56 @@
+%% Copyright 2025 Intel Corporation
+%% SPDX-License-Identifier: Apache-2.0
+
+classDiagram
+    class VerifiableDatasetInfo {
+        +label: str
+        +data_sources: DataSource[] 
+        +metadata: dict[str, str] 
+        +root_hash: HASH 
+
+        +verify_dataset(root_hash: HASH)
+        +verify_single_file(file_path: str, file_hash: HASH)
+        +to_json() str
+        +from_json(json_str: str) VerifiableDatasetInfo
+    }
+
+    class DataSource {
+        <<abstract>>
+        +name: str 
+        +type: DataSourceType 
+        +compute_file_hash(path: str) str
+        +enumerate_files() Generator~str~
+        +read_blob(path: str) bytes
+        +from_dict(ds_dict: dict) DataSource
+        +is_valid_hash_function(func) bool
+        +to_dict() dict
+    }
+
+    class LocalDataSource {
+        +base_path: str
+        ...
+    }
+
+    class S3DataSource {
+        +uri: str 
+        +endpoint: str
+        ...
+    }
+
+    class AzureBlobDataSource {
+        +name: str 
+        +container_string: str 
+        +folder_prefix: str
+        ...
+    }
+
+    VerifiableDatasetInfo "1" o-- "*" DataSource
+    DataSource <|-- LocalDataSource
+    DataSource <|-- S3DataSource
+    DataSource <|-- AzureBlobDataSource
+
+    style VerifiableDatasetInfo fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style DataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style LocalDataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style S3DataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style AzureBlobDataSource fill:#FFFFE0,stroke:#000,stroke-width:1px

docs/mermaid/verifiable_image_folder.mmd

@@ -0,0 +1,50 @@
+%% Copyright 2025 Intel Corporation
+%% SPDX-License-Identifier: Apache-2.0
+
+classDiagram
+    class torch_utils_data_Dataset {
+        +__len__() int
+        +__getitem__(index: int) Any
+    }
+
+    class VerifiableDatasetInfo {
+        +verify_dataset(root_hash: HASH)
+        +verify_single_file(file_path: str, file_hash: HASH)
+        +from_json(json_str: str) VerifiableDatasetInfo
+    }
+
+    class VerifiableMapStyleDataset {
+        <<abstract>>
+        +__len__() int
+        +__getitem__(index: int) Any
+        +create_datasets() void*
+    }
+
+    class VerifiableImageFolder {
+        +__len__() int
+        +__getitem__(index: int) Any
+        +create_datasets() void
+    }
+
+    class FolderDataset {
+        <<abstract>>
+        +__len__() int
+        +__getitem__(index: int) Any
+        +load_file(file_path: str) void*
+    }
+
+    class ImageFolder {
+        +__len__() int
+        +__getitem__(index: int) Any
+        +load_file(file_path: str) void
+    }
+
+    torch_utils_data_Dataset <|.. VerifiableMapStyleDataset
+    torch_utils_data_Dataset <|.. FolderDataset
+    VerifiableMapStyleDataset o-- VerifiableDatasetInfo
+    VerifiableMapStyleDataset <|-- VerifiableImageFolder
+    VerifiableMapStyleDataset o-- FolderDataset
+    FolderDataset <|-- ImageFolder
+
+    style torch_utils_data_Dataset fill:#D3D3D3,stroke:#000,stroke-width:1px
+    style VerifiableDatasetInfo fill:#FFFFE0,stroke:#000,stroke-width:1px