You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 16, 2023. It is now read-only.
PyGrid is a peer-to-peer network of data owners and data scientists who can collectively train AI models using [PySyft](https://github.com/OpenMined/PySyft/). PyGrid is also the central server for conducting both model-centric and data-centric federated learning.
6
6
7
-
_**A quick note about PySyft 0.3.x:** Currently, PyGrid is designed to work with the PySyft 0.2.x product line only. We are working on support for 0.3.x and hope to have this released by early 2021. Thanks for your patience!_
7
+
You may control PyGrid via our user-interface, [PyGrid Admin](https://github.com/OpenMined/pygrid-admin).
8
8
9
9
## Architecture
10
10
11
11
PyGrid platform is composed by three different components.
12
12
13
-
-**Network** - A Flask-based application used to manage, monitor, control, and route instructions to various PyGrid Nodes.
14
-
-**Node** - A Flask-based application used to store private data and models for federated learning, as well as to issue instructions to various PyGrid Workers.
15
-
-**Worker** - An emphemeral instance, managed by a PyGrid Node, that is used to compute data.
13
+
-**Network** - A Flask-based application used to manage, monitor, control, and route instructions to various PyGrid Domains.
14
+
-**Domain** - A Flask-based application used to store private data and models for federated learning, as well as to issue instructions to various PyGrid Workers.
15
+
-**Worker** - An emphemeral instance, managed by a PyGrid Domain, that is used to compute data.
16
16
17
17
## Use Cases
18
18
@@ -30,41 +30,41 @@ Model-centric FL is when the model is hosted in PyGrid. This is really useful wh
30
30
4. Once training is completed, a "diff" is generated between the new and the original state of the model
31
31
5. The diff is reported back to PyGrid and it's averaged into the model
32
32
33
-
This takes place potentially with hundreds, or thousands of devices simultaneously. **For model-centric federated learning, you only need to run a Node. Networks and Workers are irrelevant for this specific use-case.**
33
+
This takes place potentially with hundreds, or thousands of devices simultaneously. **For model-centric federated learning, you only need to run a Domain. Networks and Workers are irrelevant for this specific use-case.**
34
34
35
35
_Note:_ For posterity sake, we previously used to refer to this process as "static federated learning".
Data-centric FL is the same problem as model-centric FL, but from the opposite perspective. The most likely scenario for data-centric FL is where a person or organization has data they want to protect in PyGrid (instead of hosting the model, they host data). This would allow a data scientist who is not the data owner, to make requests for training or inference against that data. The following workflow will take place:
42
42
43
-
1. A data scientist searches for data they would like to train on (they can search either an individual Node, or a Network of Nodes)
43
+
1. A data scientist searches for data they would like to train on (they can search either an individual Domain, or a Network of Domains)
44
44
2. Once the data has been found, they may write a training plan and optionally pre-train a model
45
-
3. The training plan and model are sent to the PyGrid Node in the form of a job request
46
-
4. The PyGrid Node will gather the appropriate data from its database and send the data, the model, and the training plan to a Worker for processing
45
+
3. The training plan and model are sent to the PyGrid Domain in the form of a job request
46
+
4. The PyGrid Domain will gather the appropriate data from its database and send the data, the model, and the training plan to a Worker for processing
47
47
5. The Worker performs the plan on the model using the data
48
-
6. The result is returned to the Node
48
+
6. The result is returned to the Domain
49
49
7. The result is returned to the data scientist
50
50
51
51
For the last step, we're working on adding the capability for privacy budget tracking to be applied that will allow a data owner to "sign off" on whether or not a trained model should be released.
52
52
53
53
_Note:_ For posterity sake, we previously used to refer to this process as "dynamic federated learning".
54
54
55
-
**Node-only data-centric FL**
55
+
**Domain-only data-centric FL**
56
56
57
-
Technically speaking, it isn't required to run a Network when performing data-centric federated learning. Alternatively, as a data owner, you may opt to only run a Node, but participate in a Network hosted by someone else. The Network host will not have access to your data.
57
+
Technically speaking, it isn't required to run a Network when performing data-centric federated learning. Alternatively, as a data owner, you may opt to only run a Domain, but participate in a Network hosted by someone else. The Network host will not have access to your data.
Many times you will wat to use a Network to allow multiple Nodes to be connected together. As a data owner, it's not strictly necessary to own and operate mulitple Nodes. PyGrid doesn't prescribe one way to organize Nodes and Networks, but we expose these applications to allow you and various related stakeholders to make the correct decision about your infrastructure needs.
63
+
Many times you will wat to use a Network to allow multiple Domains to be connected together. As a data owner, it's not strictly necessary to own and operate mulitple Domains. PyGrid doesn't prescribe one way to organize Domains and Networks, but we expose these applications to allow you and various related stakeholders to make the correct decision about your infrastructure needs.
Currently, we suggest two ways to run PyGrid locally: Docker and manually running from source. With Docker, we can organize all the services we'd like to use and then boot them all in one command. With manually running from source, we have to run them as separate tasks.
70
70
@@ -74,14 +74,14 @@ To install Docker, just follow the [docker documentation](https://docs.docker.co
74
74
75
75
#### 1. Setting the your hostfile
76
76
77
-
Before start the grid platform locally using Docker, we need to set up the domain names used by the bridge network. In order to use these nodes from outside of the containers context, you should add the following domain names on your `/etc/hosts`
77
+
Before start the grid platform locally using Docker, we need to set up the domain names used by the bridge network. In order to use these Domains from outside of the containers context, you should add the following domain names on your `/etc/hosts`
78
78
79
79
```
80
80
127.0.0.1 network
81
-
127.0.0.1 bob
82
81
127.0.0.1 alice
83
-
127.0.0.1 bill
84
-
127.0.0.1 james
82
+
127.0.0.1 bob
83
+
127.0.0.1 charlie
84
+
127.0.0.1 dan
85
85
```
86
86
87
87
Note that you're not restricted to running 4 nodes and a network. You could instead run just a single node if you'd like - this is often all you need for model-centric federated learning. For the sake of our example, we'll use the network running 4 nodes underneath but you're welcome to modify it to your needs.
@@ -90,8 +90,9 @@ Note that you're not restricted to running 4 nodes and a network. You could inst
90
90
91
91
The latest PyGrid Network and Node images are also available on the Docker Hub.
> First install [`poetry`](https://python-poetry.org/docs/) and run `poetry install` in `apps/node`
132
+
122
133
To start the PyGrid Node manually, run:
123
134
124
135
```
@@ -132,15 +143,15 @@ You can pass the arguments or use environment variables to set the network confi
132
143
133
144
-`-h, --help` - Shows the help message and exit
134
145
-`-p [PORT], --port [PORT]` - Port to run server on (default: 5000)
135
-
-`--host [HOST]` - The Network host
146
+
-`--host [HOST]` - The Node host
136
147
-`--num_replicas [NUM]` - The number of replicas to provide fault tolerance to model hosting
137
148
-`--id [ID]` - The ID of the Node
138
149
-`--start_local_db` - If this flag is used a SQLAlchemy DB URI is generated to use a local db
139
150
140
151
**Environment Variables**
141
152
142
-
-`GRID_NETWORK_PORT` - Port to run server on
143
-
-`GRID_NETWORK_HOST` - The Network host
153
+
-`GRID_NODE_PORT` - Port to run server on
154
+
-`GRID_NODE_HOST` - The Node host
144
155
-`NUM_REPLICAS` - Number of replicas to provide fault tolerance to model hosting
145
156
-`DATABASE_URL` - The Node database URL
146
157
-`SECRET_KEY` - The secret key
@@ -151,15 +162,15 @@ To start the PyGrid Network manually, run:
151
162
152
163
```
153
164
cd apps/network
154
-
./run.sh --port 5000 --start_local_db
165
+
./run.sh --port 7000 --start_local_db
155
166
```
156
167
157
168
You can pass the arguments or use environment variables to set the network configs.
158
169
159
170
**Arguments**
160
171
161
172
-`-h, --help` - Shows the help message and exit
162
-
-`-p [PORT], --port [PORT]` - Port to run server on (default: 5000)
173
+
-`-p [PORT], --port [PORT]` - Port to run server on (default: 7000)
163
174
-`--host [HOST]` - The Network host
164
175
-`--start_local_db` - If this flag is used a SQLAlchemy DB URI is generated to use a local db
165
176
@@ -170,13 +181,18 @@ You can pass the arguments or use environment variables to set the network confi
170
181
-`DATABASE_URL` - The Network database URL
171
182
-`SECRET_KEY` - The secret key
172
183
184
+
## Deployment & CLI
185
+
186
+
[Please check the instruction for deployment and CLI here.](deployment.md)
187
+
188
+
173
189
## Contributing
174
190
175
191
If you're interested in contributing, check out our [Contributor Guidelines](CONTRIBUTING.md).
176
192
177
193
## Support
178
194
179
-
For support in using this library, please join the **#lib_pygrid** Slack channel. If you’d like to follow along with any code changes to the library, please join the **#code_pygrid** Slack channel. [Click here to join our Slack community!](https://slack.openmined.org)
195
+
For support in using this library, please join the **#support** Slack channel. [Click here to join our Slack community!](https://slack.openmined.org)
0 commit comments