Initial work for enabling interoperability with Flower framework

[kminhta]

Summary

Enable interoperability between Flower and OpenFL

Motivation

• Improvements toward federated learning standards
• Promote OpenFL to Flower's developer community
• Enable Flower's development community to leverage TEE (Intel(R) SGX) and Intel(R) Tiber(R) Secure Federated AI

Proposal

Enable Flower workloads to run through OpenFL's Task Runner API. Consider a standard federation with a centralized aggregation server and distributed collaborator client sites. At the server Flower server components will communicate locally with OpenFL server components via the FlowerInteropClient (local gRPC client). At the client sites, Flower client components will communicate locally with OpenFL client components via the FlowerInteropServer (local gRPC server). The message across the network will be managed by OpenFL.

This will require:

• a new RPC added to the Aggregator service for relaying interoperability messages with associated metadata
• new local gRPC components (FlowerInteropClient and FlowerInteropServer ) and that can relay messages between the Flower and OpenFL components
• a new OpenFL Connector component that will manage the FlowerInteropClient and Flower server components
• a Flower-specific Task Runner that will manage the FlowerInteropServer and Flower client components
• a Flower-specific data loader to allow the Flower components to properly ingest data
• extend aggregator to initialize and manage the Connector
• extend collaborator to run the Flower Task Runner and initialize the FlowerInteropServer
• misc updates to existing components to allow the experiment to run in an "interopability mode" to bypass need to create TensorDB, initial model parameters, etc. since Flower will be responsible for aggregation, loss calculation, etc. in this implementation.

How it will work

• a Flower workspace will contain an updated plan.yaml that includes the Connector and the Flower Task Runner and Data Loader.
• in the Flower workspace, under the ./src directory, a user will add the source code required to run a Flower experiment (i.e. app-pytorch)
• a user will set up the workspace as normal (i.e. certifications, collaborator creation, etc.)
• fx aggregator start will start the aggregator server, the SuperLink, and the ServerApp
• fx collaborator start -n collaboratorX will start the collaborator client. The collaborator call the GetTask() which will pull in a single task start_client_adapter which will instruct the Task Runner to start the SuperNode and ClientApp
• the Flower experiment will run
• once the Flower experiment completes, the ServerApp will shutoff, which will trigger the FlowerInteropClient to amend a flag to end the experiment as a metadata in the InteropMessage. This will signal the collaborator to shutdown the ClientApp and SuperNode, then call TaskResults which is intended to signal the end of the OpenFL round. OpenFL will only run for 1 round (the Flower will run for as many rounds as indicated in the Flower app's pyproject.toml. This will start shutdown of all components and thus complete the experiment

[Sequence Diagram to come]

We will split this into Two Phases for now

Phase 1 [Completed]
Contain the majority of workspace related changes inside a standalone workspace flower-app-pytorch #1433

Phase 2 [In Progress]
Begin to migrate extended methods and added scripts to OpenFL core-level components

[teoparvanov]

@kta-intel , this looks great, thanks for sharing the vision and write-up! As discussed offline, could you rename the local grpc client/server resp. to interop client/server in the text and in the diagrams?