Skip to content

Commit 3e9cfd2

Browse files
ngraylunamdlinville
ngrayluna
and
mdlinville
authored
Sweeps: Fixes code snippet(s), condenses page (#1518)
Description ----------- Started off with a simple code check, evolved to rewriting most of this page. Related issues ----------- - Fixes https://wandb.atlassian.net/browse/DOCS-1695 <!-- preview-links-comment --> 📄 **[View preview links for changed pages](#1518 (comment) --------- Co-authored-by: Matt Linville <matt.linville@wandb.com>
1 parent f92113d commit 3e9cfd2

1 file changed

Lines changed: 107 additions & 151 deletions

File tree

‎content/en/guides/models/sweeps/add-w-and-b-to-your-code.md‎

Lines changed: 107 additions & 151 deletions
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,27 @@ title: Add W&B (wandb) to your code
88
weight: 2
99
---
1010

11-
There are numerous ways to add the W&B Python SDK to your script or notebook. This section provides a "best practice" example that shows how to integrate the W&B Python SDK into your own code.
11+
This guide provides recommendations on how to integrate W&B into your Python training script or notebook.
1212

13-
### Original training script
13+
## Original training script
1414

15-
Suppose you have the following code in a Python script. We define a function called `main` that mimics a typical training loop. For each epoch, the accuracy and loss is computed on the training and validation data sets. The values are randomly generated for the purpose of this example.
15+
Suppose you have a Python script that trains a model (see below). Your goal is to find the hyperparameters that maxmimizes the validation accuracy(`val_acc`).
1616

17-
We defined a dictionary called `config` where we store hyperparameters values. At the end of the cell, we call the `main` function to execute the mock training code.
17+
In your Python script, you define two functions: `train_one_epoch` and `evaluate_one_epoch`. The `train_one_epoch` function simulates training for one epoch and returns the training accuracy and loss. The `evaluate_one_epoch` function simulates evaluating the model on the validation data set and returns the validation accuracy and loss.
18+
19+
You define a configuration dictionary (`config`) that contains hyperparameter values such as the learning rate (`lr`), batch size (`batch_size`), and number of epochs (`epochs`). The values in the configuration dictionary control the training process.
20+
21+
Next you define a function called `main` that mimics a typical training loop. For each epoch, the accuracy and loss is computed on the training and validation data sets.
22+
23+
{{< alert >}}
24+
This code is a mock training script. It does not train a model, but simulates the training process by generating random accuracy and loss values. The purpose of this code is to demonstrate how to integrate W&B into your training script.
25+
{{< /alert >}}
1826

1927
```python
2028
import random
2129
import numpy as np
2230

23-
def train_one_epoch(epoch, lr, bs):
31+
def train_one_epoch(epoch, lr, batch_size):
2432
acc = 0.25 + ((epoch / 30) + (random.random() / 10))
2533
loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
2634
return acc, loss
@@ -31,129 +39,127 @@ def evaluate_one_epoch(epoch):
3139
return acc, loss
3240

3341
# config variable with hyperparameter values
34-
config = {"lr": 0.0001, "bs": 16, "epochs": 5}
42+
config = {"lr": 0.0001, "batch_size": 16, "epochs": 5}
3543

3644
def main():
37-
# Note that we define values from `wandb.Run.config`
38-
# instead of defining hard values
3945
lr = config["lr"]
40-
bs = config["bs"]
46+
batch_size = config["batch_size"]
4147
epochs = config["epochs"]
4248

4349
for epoch in np.arange(1, epochs):
44-
train_acc, train_loss = train_one_epoch(epoch, lr, bs)
50+
train_acc, train_loss = train_one_epoch(epoch, lr, batch_size)
4551
val_acc, val_loss = evaluate_one_epoch(epoch)
4652

4753
print("epoch: ", epoch)
4854
print("training accuracy:", train_acc, "training loss:", train_loss)
49-
print("validation accuracy:", val_acc, "training loss:", val_loss)
55+
print("validation accuracy:", val_acc, "validation loss:", val_loss)
56+
57+
if __name__ == "__main__":
58+
main()
5059
```
5160

52-
### Training script with W&B Python SDK
61+
In the next section, you will add W&B to your Python script to track hyperparameters and metrics during training. You want to use W&B to find the best hyperparameters that maximize the validation accuracy (`val_acc`).
62+
5363

54-
The following code examples demonstrate how to add the W&B Python SDK into your
55-
code. If you start W&B Sweep jobs in the CLI, you will want to explore the CLI
56-
tab. If you start W&B Sweep jobs within a Jupyter notebook or Python script,
57-
explore the Python SDK tab.
64+
## Training script with W&B Python SDK
5865

59-
{{< tabpane text=true >}} {{% tab header="Python script or notebook" %}} To
60-
create a W&B Sweep, we added the following to the code example:
66+
How you integrate W&B to your Python script or notebook depends on how you manage sweeps. You can start a sweep job within a Python notebook or script or from the command line.
6167

62-
1. Import the W&B Python SDK.
63-
2. Create a dictionary object where the key-value pairs define the sweep configuration. In the proceeding example, the batch size (`batch_size`), epochs (`epochs`), and the learning rate (`lr`) hyperparameters are varied during each sweep. For more information, see [Define sweep configuration]({{< relref "/guides/models/sweeps/define-sweep-configuration/" >}}).
64-
3. Pass the sweep configuration dictionary to [`wandb.sweep()`]({{< relref "/ref/python/sdk/functions/sweep.md" >}}). This initializes the sweep. This returns a sweep ID (`sweep_id`). For more information, see [Initialize sweeps]({{< relref "./initialize-sweeps.md" >}}).
65-
4. Use the [`wandb.init()`]({{< relref "/ref/python/sdk/functions/init.md" >}}) API to generate a background process to sync and log data as a [W&B Run]({{< relref "/ref/python/sdk/classes/run.md" >}}).
66-
5. (Optional) define values from `wandb.config` instead of defining hard coded values.
67-
6. Log the metric you want to optimize with [`wandb.Run.log()`]({{< relref "/ref/python/sdk/classes/run.md/#method-runlog" >}}). You must log the metric defined in your configuration. Within the configuration dictionary (`sweep_configuration` in this example), you define the sweep to maximize the `val_acc` value.
68-
7. Start the sweep with the [`wandb.agent`]({{< relref "/ref/python/sdk/functions/agent.md" >}}) API call. Provide the sweep ID and the name of the function the sweep will execute (`function=main`), and specify the maximum number of runs to try to four (`count=4`). For more informationp, see [Start sweep agents]({{< relref "./start-sweep-agents.md" >}}).
68+
{{< tabpane text=true >}}
69+
{{% tab header="Python script or notebook" %}}
6970

71+
Add the following to your Python script:
72+
73+
1. Create a dictionary object where the key-value pairs define a [sweep configuration]({{< relref "/guides/models/sweeps/define-sweep-configuration/" >}}). The sweep configuration defines the hyperparameters you want W&B to explore on your behalf along with the metric you want to optimize. Continuing from the previous example, the batch size (`batch_size`), epochs (`epochs`), and the learning rate (`lr`) are the hyperparameters to vary during each sweep. You want to maximize the accuracy of the validation score so you set `"goal": "maximize"` and the name of the variable you want to optimize for, in this case `val_acc` (`"name": "val_acc"`).
74+
2. Pass the sweep configuration dictionary to [`wandb.sweep()`]({{< relref "/ref/python/sdk/functions/sweep.md" >}}). This initializes the sweep and returns a sweep ID (`sweep_id`). For more information, see [Initialize sweeps]({{< relref "./initialize-sweeps.md" >}}).
75+
3. At the top of your script, import the W&B Python SDK (`wandb`).
76+
4. Within your `main` function, use the [`wandb.init()`]({{< relref "/ref/python/sdk/functions/init.md" >}}) API to generate a background process to sync and log data as a [W&B Run]({{< relref "/ref/python/sdk/classes/run.md" >}}). Pass the project name as a parameter to the `wandb.init()` method. If you do not pass a project name, W&B uses the default project name.
77+
5. Fetch the hyperparameter values from the `wandb.Run.config` object. This allows you to use the hyperparameter values defined in the sweep configuration dictionary instead of hard coded values.
78+
6. Log the metric you are optimizing for to W&B using [`wandb.Run.log()`]({{< relref "/ref/python/sdk/classes/run.md/#method-runlog" >}}). You must log the metric defined in your configuration. For example, if you define the metric to optimize as `val_acc`, you must log `val_acc`. If you do not log the metric, W&B does not know what to optimize for. Within the configuration dictionary (`sweep_configuration` in this example), you define the sweep to maximize the `val_acc` value.
79+
7. Start the sweep with [`wandb.agent()`]({{< relref "/ref/python/sdk/functions/agent.md" >}}). Provide the sweep ID and the name of the function the sweep will execute (`function=main`), and specify the maximum number of runs to try to four (`count=4`).
80+
81+
82+
Putting this all together, your script might look similar to the following:
7083

7184
```python
72-
import wandb
85+
import wandb # Import the W&B Python SDK
7386
import numpy as np
7487
import random
88+
import argparse
7589

76-
77-
# Define training function that takes in hyperparameter
78-
# values from `wandb.Run.config` and uses them to train a
79-
# model and return the metrics
80-
def train_one_epoch(epoch, lr, bs):
90+
def train_one_epoch(epoch, lr, batch_size):
8191
acc = 0.25 + ((epoch / 30) + (random.random() / 10))
8292
loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
8393
return acc, loss
8494

85-
8695
def evaluate_one_epoch(epoch):
8796
acc = 0.1 + ((epoch / 20) + (random.random() / 10))
8897
loss = 0.25 + (1 - ((epoch - 1) / 10 + random.random() / 6))
8998
return acc, loss
9099

91-
92-
# Define a sweep config dictionary
93-
sweep_configuration = {
94-
"method": "random",
95-
"name": "sweep",
96-
"metric": {"goal": "maximize", "name": "val_acc"},
97-
"parameters": {
98-
"batch_size": {"values": [16, 32, 64]},
99-
"epochs": {"values": [5, 10, 15]},
100-
"lr": {"max": 0.1, "min": 0.0001},
101-
},
102-
}
103-
104-
# (Optional) Provide a name for the project.
105-
project = "my-first-sweep"
106-
107-
def main():
108-
# Use the `with` context manager statement to automatically end the run.
109-
# This is equivalent to using `run.finish()` at the end of each run
100+
def main(args=None):
101+
# When called by sweep agent, args will be None,
102+
# so we use the project from sweep config
103+
project = args.project if args else None
104+
110105
with wandb.init(project=project) as run:
111-
112-
# This code fetches the hyperparameter values from `wandb.Run.config`
113-
# instead of defining them explicitly
106+
# Fetches the hyperparameter values from `wandb.Run.config` object
114107
lr = run.config["lr"]
115-
bs = run.config["batch_size"]
108+
batch_size = run.config["batch_size"]
116109
epochs = run.config["epochs"]
117110

118111
# Execute the training loop and log the performance values to W&B
119112
for epoch in np.arange(1, epochs):
120-
train_acc, train_loss = train_one_epoch(epoch, lr, bs)
113+
train_acc, train_loss = train_one_epoch(epoch, lr, batch_size)
121114
val_acc, val_loss = evaluate_one_epoch(epoch)
122-
123115
run.log(
124116
{
125117
"epoch": epoch,
126118
"train_acc": train_acc,
127119
"train_loss": train_loss,
128-
"val_acc": val_acc,
120+
"val_acc": val_acc, # Metric optimized
129121
"val_loss": val_loss,
130122
}
131123
)
132124

133-
134125
if __name__ == "__main__":
126+
parser = argparse.ArgumentParser()
127+
parser.add_argument("--project", type=str, default="sweep-example", help="W&B project name")
128+
args = parser.parse_args()
129+
130+
# Define a sweep config dictionary
131+
sweep_configuration = {
132+
"method": "random",
133+
"name": "sweep",
134+
# Metric that you want to optimize
135+
# For example, if you want to maximize validation
136+
# accuracy set "goal": "maximize" and the name of the variable
137+
# you want to optimize for, in this case "val_acc"
138+
"metric": {
139+
"goal": "maximize",
140+
"name": "val_acc"
141+
},
142+
"parameters": {
143+
"batch_size": {"values": [16, 32, 64]},
144+
"epochs": {"values": [5, 10, 15]},
145+
"lr": {"max": 0.1, "min": 0.0001},
146+
},
147+
}
148+
135149
# Initialize the sweep by passing in the config dictionary
136-
sweep_id = wandb.sweep(sweep=sweep_configuration, project=project)
150+
sweep_id = wandb.sweep(sweep=sweep_configuration, project=args.project)
137151

138152
# Start the sweep job
139153
wandb.agent(sweep_id, function=main, count=4)
140-
141154
```
142155

143-
{{% alert %}} The preceding code snippet shows how to initialize a
144-
[`wandb.init()`]({{< relref "/ref/python/sdk/functions/init.md" >}}) API within a `with`
145-
context manager statement to generate a background process to sync and log data
146-
as a [W&B Run]({{< relref "/ref/python/sdk/classes/run.md" >}}). This ensures the run is
147-
properly terminated after uploading the logged values. An alternative approach
148-
is to call `wandb.init()` and `wandb.Run.finish()` at the beginning and end of the
149-
training script, respectively.
150-
{{% /alert %}}
156+
151157

152158
{{% /tab %}} {{% tab header="CLI" %}}
153159

154-
To create a W&B Sweep, we first create a YAML configuration file. The
155-
configuration file contains the hyperparameters we want the sweep to explore. In
156-
the proceeding example, the batch size (`batch_size`), epochs (`epochs`), and
160+
Create a YAML configuration file with your sweep configuration. The
161+
configuration file contains the hyperparameters you want the sweep to explore. In
162+
the following example, the batch size (`batch_size`), epochs (`epochs`), and
157163
the learning rate (`lr`) hyperparameters are varied during each sweep.
158164

159165
```yaml
@@ -179,13 +185,13 @@ For more information on how to create a W&B Sweep configuration, see [Define swe
179185
You must provide the name of your Python script for the `program` key
180186
in your YAML file.
181187

182-
Next, we add the following to the code example:
188+
Next, add the following to the code example:
183189

184190
1. Import the W&B Python SDK (`wandb`) and PyYAML (`yaml`). PyYAML is used to read in our YAML configuration file.
185191
2. Read in the configuration file.
186-
3. Use the [`wandb.init()`]({{< relref "/ref/python/sdk/functions/init.md" >}}) API to generate a background process to sync and log data as a [W&B Run]({{< relref "/ref/python/sdk/classes/run.md" >}}). We pass the config object to the config parameter.
192+
3. Use the [`wandb.init()`]({{< relref "/ref/python/sdk/functions/init.md" >}}) API to generate a background process to sync and log data as a [W&B Run]({{< relref "/ref/python/sdk/classes/run.md" >}}). Pass the config object to the config parameter.
187193
4. Define hyperparameter values from `wandb.Run.config` instead of using hard coded values.
188-
5. Log the metric we want to optimize with [`wandb.Run.log()`]({{< relref "/ref/python/sdk/classes/run.md/#method-runlog" >}}). You must log the metric defined in your configuration. Within the configuration dictionary (`sweep_configuration` in this example) we defined the sweep to maximize the `val_acc` value.
194+
5. Log the metric you want to optimize with [`wandb.Run.log()`]({{< relref "/ref/python/sdk/classes/run.md/#method-runlog" >}}). You must log the metric defined in your configuration. Within the configuration dictionary (`sweep_configuration` in this example) you define the sweep to maximize the `val_acc` value.
189195

190196
```python
191197
import wandb
@@ -194,7 +200,7 @@ import random
194200
import numpy as np
195201
196202
197-
def train_one_epoch(epoch, lr, bs):
203+
def train_one_epoch(epoch, lr, batch_size):
198204
acc = 0.25 + ((epoch / 30) + (random.random() / 10))
199205
loss = 0.2 + (1 - ((epoch - 1) / 10 + random.random() / 5))
200206
return acc, loss
@@ -246,7 +252,7 @@ wandb sweep --project sweep-demo-cli config.yaml
246252
This returns a sweep ID. For more information on how to initialize sweeps, see
247253
[Initialize sweeps]({{< relref "./initialize-sweeps.md" >}}).
248254

249-
Copy the sweep ID and replace `sweepID` in the proceeding code snippet to start
255+
Copy the sweep ID and replace `sweepID` in the following code snippet to start
250256
the sweep job with the [`wandb agent`]({{< relref "/ref/cli/wandb-agent.md" >}})
251257
command:
252258

@@ -258,84 +264,34 @@ For more information, see [Start sweep jobs]({{< relref "./start-sweep-agents.md
258264

259265
{{% /tab %}} {{< /tabpane >}}
260266

261-
## Consideration when logging metrics
262267

263-
Be sure to log the sweep's metric to W&B explicitly. Do not log metrics for your sweep inside a subdirectory.
264-
265-
For example, consider the following pseudocode. A user wants to log the validation loss (`"val_loss": loss`). First they pass the values into a dictionary. However, the dictionary passed to `wandb.Run.log()` does not explicitly access the key-value pair in the dictionary:
268+
{{% alert title="Logging metrics to W&B in a sweep" %}}
269+
You must log the metric you define and are optimizing for in both your sweep configuration and with `wandb.Run.log()`. For example, if you define the metric to optimize as `val_acc` within your sweep configuration, you must also log `val_acc` to W&B. If you do not log the metric, W&B does not know what to optimize for.
266270

267271
```python
268-
# Import the W&B Python Library and log into W&B
269-
import wandb
270-
import random
271-
272-
def train():
273-
# Simulate training and validation metrics
274-
offset = random.random() / 5
275-
epoch = 5 # Simulate an epoch value
276-
acc = 1 - 2**-epoch - random.random() / epoch - offset
277-
loss = 2**-epoch + random.random() / epoch + offset
278-
return loss, acc
279-
280-
281-
def main():
282-
with wandb.init(entity="<entity>", project="my-first-sweep") as run:
283-
val_loss, val_acc = train()
284-
# Incorrect. You must explicitly access the
285-
# key-value pair in the dictionary
286-
# See next code block to see how to correctly log metrics
287-
run.log({"val_loss": val_loss, "val_acc": val_acc})
288-
289-
sweep_configuration = {
290-
"method": "random",
291-
"metric": {"goal": "minimize", "name": "val_loss"},
292-
"parameters": {
293-
"x": {"max": 0.1, "min": 0.01},
294-
"y": {"values": [1, 3, 7]},
295-
},
296-
}
297-
298-
# Initialize the sweep with the configuration dictionary
299-
sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")
300-
301-
# Start the sweep job
302-
wandb.agent(sweep_id, function=main, count=10)
272+
with wandb.init() as run:
273+
val_loss, val_acc = train()
274+
run.log(
275+
{
276+
"val_loss": val_loss,
277+
"val_acc": val_acc
278+
}
279+
)
303280
```
304281

305-
Instead, explicitly access the key-value pair within the Python dictionary. For example, the following code specifies the key-value pair when you pass the dictionary to the `wandb.Run.log()` method:
306-
307-
```python title="train.py"
308-
# Import the W&B Python Library and log into W&B
309-
import wandb
310-
import random
311-
282+
The following is an incorrect example of logging the metric to W&B. The metric that is optimized for in the sweep configuration is `val_acc`, but the code logs `val_acc` within a nested dictionary under the key `validation`. You must log the metric directly, not within a nested dictionary.
312283

313-
def train():
314-
offset = random.random() / 5
315-
acc = 1 - 2**-epoch - random.random() / epoch - offset
316-
loss = 2**-epoch + random.random() / epoch + offset
317-
318-
return loss, acc
319-
320-
321-
def main():
322-
with wandb.init(entity="<entity>", project="my-first-sweep") as run:
323-
# Correct. Explicitly access the key-value pair in the dictionary
324-
# when logging metrics
325-
val_loss, val_acc = train()
326-
run.log({"val_loss": val_loss, "val_acc": val_acc})
327-
328-
329-
sweep_configuration = {
330-
"method": "random",
331-
"metric": {"goal": "minimize", "name": "val_loss"},
332-
"parameters": {
333-
"x": {"max": 0.1, "min": 0.01},
334-
"y": {"values": [1, 3, 7]},
335-
},
336-
}
337-
338-
sweep_id = wandb.sweep(sweep=sweep_configuration, project="my-first-sweep")
339-
340-
wandb.agent(sweep_id, function=main, count=10)
284+
```python
285+
with wandb.init() as run:
286+
val_loss, val_acc = train()
287+
run.log(
288+
{
289+
"validation": {
290+
"val_loss": val_loss,
291+
"val_acc": val_acc
292+
}
293+
}
294+
)
341295
```
296+
297+
{{% /alert %}}

0 commit comments

Comments
 (0)