Skip to content

Commit 64f9acb

Browse files
ngraylunangrayluna
authored
Tables: Table log modes (#1335)
Dominc Phan did an awesome job creating this first draft of table modes. Used that as a starting point and created this branch. Doc preview: https://tables-incremental-mutable-d.docodile.pages.dev/guides/models/tables/log_tables/ Jira ticket: https://wandb.atlassian.net/browse/DOCS-1508 STATUS: Ready for tech writer review
1 parent 524b75b commit 64f9acb

1 file changed

Lines changed: 271 additions & 0 deletions

File tree

‎content/guides/models/tables/log_tables.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
---
2+
title: Log tables
3+
weight: 2
4+
---
5+
6+
Visualize and log tabular data with W&B Tables. A W&B Table is a two-dimensional grid of data where each column has a single type of data. Each row represents one or more data points logged to a W&B [run]({{< relref "/guides/models/track/runs/" >}}). W&B Tables support primitive and numeric types, as well as nested lists, dictionaries, and rich media types.
7+
8+
W&B Tables are a specialized [data type]({{< relref "/ref/python/data-types/" >}}) in W&B that are logged internally in W&B as [artifacts]({{< relref "/guides/core/artifacts/" >}}) objects.
9+
10+
You [create and log table objects]({{< relref "#create-and-log-a-new-table" >}}) using the W&B Python SDK. When you create a table object, you specify the columns and data for the table and a [mode]({{< relref "#table-logging-modes" >}}). The mode determines how the table is logged and updated during your ML experiments.
11+
12+
13+
## Create and log a table
14+
15+
16+
1. Initialize a new run with `wandb.init()`.
17+
2. Create a table object with the [`wandb.Table`]({{< relref "/ref/python/data-types/table" >}}) Class. Specify the columns and data for the table for the `columns` and `data` parameters, respectively. Though optional, it is recommended to set the `log_mode` parameter to one of the three modes: `IMMUTABLE`, `MUTABLE`, or `INCREMENTAL`. The default mode is `IMMUTABLE`. See [Table Logging Modes]({{< relref "#table-logging-modes" >}}) in the next section for more information.
18+
3. Log the table to W&B with `run.log()`.
19+
20+
The following example shows how to create and log a table with two columns, `a` and `b`, and two rows of data, `["a1", "b1"]` and `["a2", "b2"]`:
21+
22+
```python
23+
import wandb
24+
25+
# Start a new run
26+
run = wandb.init(project="table-demo")
27+
28+
# Create a table object with two columns and two rows of data
29+
my_table = wandb.Table(
30+
columns=["a", "b"],
31+
data=[["a1", "b1"], ["a2", "b2"]],
32+
log_mode="IMMUTABLE"
33+
)
34+
35+
# Log the table to W&B
36+
run.log({"Table Name": my_table})
37+
38+
# Finish the run
39+
run.finish()
40+
```
41+
42+
## Logging modes
43+
44+
The [`wandb.Table`]({{< relref "/ref/python/data-types/table" >}}) `log_mode` parameter determines how a table is logged and updated during your ML experiments. The `log_mode` parameter accepts one of three arguments: `IMMUTABLE`, `MUTABLE`, and `INCREMENTAL`. Each mode has different implications for how a table is logged, how it can be modified, and how it is rendered in the W&B App.
45+
46+
The following describes the three logging modes, the high-level differences, and common use case for each mode:
47+
48+
| Mode | Definition | Use Cases | Benefits |
49+
| ----- | ---------- | ---------- | ----------|
50+
| `IMMUTABLE` | Once a table is logged to W&B, you cannot modify it. |- Storing tabular data generated at the end of a run for further analysis | - Minimal overhead when logged at the end of a run<br>- All rows rendered in UI |
51+
| `MUTABLE` | After you log a table to W&B, you can overwrite the existing table with a new one. | - Adding columns or rows to existing tables<br>- Enriching results with new information | - Capture Table mutations<br>- All rows rendered in UI |
52+
| `INCREMENTAL` | Add batches of new rows to a table throughout the machine learning experiment. | - Adding rows to tables in batches<br> - Long-running training jobs<br>- Processing large datasets in batches<br>- Monitoring ongoing results | - View updates on UI during training<br>- Ability to step through increments |
53+
54+
The next sections show example code snippets for each mode along with considerations when to use each mode.
55+
56+
### MUTABLE mode
57+
58+
`MUTABLE` mode updates an existing table by replacing the existing table with a new one. `MUTABLE` mode is useful when you want to add new columns and rows to an existing table in a non iterative process. Within the UI, the table is rendered with all rows and columns, including the new ones added after the initial log.
59+
60+
{{% alert %}}
61+
In `MUTABLE` mode, the table object is replaced each time you log the table. Overwriting a table with a new one is computationally expensive and can be slow for large tables.
62+
{{% /alert %}}
63+
64+
The following example shows how to create a table in `MUTABLE` mode, log it, and then add new columns to it. The table object is logged three times: once with the initial data, once with the confidence scores, and once with the final predictions.
65+
66+
{{% alert %}}
67+
The following example uses a placeholder function `load_eval_data()` to load data and a placeholder function `model.predict()` to make predictions. You will need to replace these with your own data loading and prediction functions.
68+
{{% /alert %}}
69+
70+
```python
71+
import wandb
72+
import numpy as np
73+
74+
run = wandb.init(project="mutable-table-demo")
75+
76+
# Create a table object with MUTABLE logging mode
77+
table = wandb.Table(columns=["input", "label", "prediction"],
78+
log_mode="MUTABLE")
79+
80+
# Load data and make predictions
81+
inputs, labels = load_eval_data() # Placeholder function
82+
raw_preds = model.predict(inputs) # Placeholder function
83+
84+
for inp, label, pred in zip(inputs, labels, raw_preds):
85+
table.add_data(inp, label, pred)
86+
87+
# Step 1: Log initial data
88+
wandb.log({"eval_table": table}) # Log initial table
89+
90+
# Step 2: Add confidence scores (e.g. max softmax)
91+
confidences = np.max(raw_preds, axis=1)
92+
table.add_column("confidence", confidences)
93+
run.log({"eval_table": table}) # Add confidence info
94+
95+
# Step 3: Add post-processed predictions
96+
# (e.g., thresholded or smoothed outputs)
97+
post_preds = (confidences > 0.7).astype(int)
98+
table.add_column("final_prediction", post_preds)
99+
wandb.log({"eval_table": table}) # Final update with another column
100+
101+
run.finish()
102+
```
103+
104+
If you only want to add new batches of rows (no columns) incrementally like in a training loop, consider using [`INCREMENTAL` mode]({{< relref "#INCREMENTAL-mode" >}}) instead.
105+
106+
### INCREMENTAL mode
107+
108+
In incremental mode, you log batches of rows to a table during the machine learning experiment. This is ideal for monitoring long-running jobs or when working with large tables that would be inefficient to log during the run for updates. Within the UI, the table is updated with new rows as they are logged, allowing you to view the latest data without having to wait for the entire run to finish. You can also step through the increments to view the table at different points in time.
109+
110+
{{% alert %}}
111+
Run workspaces in the W&B App have a limit of 100 increments. If you log more than 100 increments, only the most recent 100 are shown in the run workspace.
112+
{{% /alert %}}
113+
114+
The following example creates a table in `INCREMENTAL` mode, logs it, and then adds new rows to it. Note that the table is logged once per training step (`step`).
115+
116+
{{% alert %}}
117+
The following example uses a placeholder function `get_training_batch()` to load data, a placeholder function `train_model_on_batch()` to train the model, and a placeholder function `predict_on_batch()` to make predictions. You will need to replace these with your own data loading, training, and prediction functions.
118+
{{% /alert %}}
119+
120+
```python
121+
import wandb
122+
123+
run = wandb.init(project="incremental-table-demo")
124+
125+
# Create a table with INCREMENTAL logging mode
126+
table = wandb.Table(columns=["step", "input", "label", "prediction"],
127+
log_mode="INCREMENTAL")
128+
129+
# Training loop
130+
for step in range(get_num_batches()): # Placeholder function
131+
# Load batch data
132+
inputs, labels = get_training_batch(step) # Placeholder function
133+
134+
# Train and predict
135+
train_model_on_batch(inputs, labels) # Placeholder function
136+
predictions = predict_on_batch(inputs) # Placeholder function
137+
138+
# Add batch data to table
139+
for input_item, label, prediction in zip(inputs, labels, predictions):
140+
table.add_data(step, input_item, label, prediction)
141+
142+
# Log the table incrementally
143+
wandb.log({"training_table": table}, step=step)
144+
145+
run.finish()
146+
```
147+
148+
Incremental logging is generally more computationally efficient than logging a new table each time (`log_mode=MUTABLE`). However, the W&B App may not render all rows in the table if you log a large number of increments. If your goal is to update and view your table data while your run is ongoing and to have all the data available for analysis, consider using two tables. One with `INCREMENTAL` log mode and one with `IMMUTABLE` log mode.
149+
150+
The following example shows how to combine `INCREMENTAL` and `IMMUTABLE` logging modes to achieve this.
151+
152+
```python
153+
import wandb
154+
155+
run = wandb.init(project="combined-logging-example")
156+
157+
# Create an incremental table for efficient updates during training
158+
incr_table = wandb.Table(columns=["step", "input", "prediction", "label"],
159+
log_mode="INCREMENTAL")
160+
161+
# Training loop
162+
for step in range(get_num_batches()):
163+
# Process batch
164+
inputs, labels = get_training_batch(step)
165+
predictions = model.predict(inputs)
166+
167+
# Add data to incremental table
168+
for inp, pred, label in zip(inputs, predictions, labels):
169+
incr_table.add_data(step, inp, pred, label)
170+
171+
# Log the incremental update (suffix with -incr to distinguish from final table)
172+
run.log({"table-incr": incr_table}, step=step)
173+
174+
# At the end of training, create a complete immutable table with all data
175+
# Using the default IMMUTABLE mode to preserve the complete dataset
176+
final_table = wandb.Table(columns=incr_table.columns, data=incr_table.data, log_mode="IMMUTABLE")
177+
run.log({"table": final_table})
178+
179+
run.finish()
180+
```
181+
182+
In this example, the `incr_table` is logged incrementally (with `log_mode="INCREMENTAL"`) during training. This allows you to log and view updates to the table as new data is processed. At the end of training, an immutable table (`final_table`) is created with all data from the incremental table. The immutable table is logged to preserve the complete dataset for further analysis and it enables you to view all rows in the W&B App.
183+
184+
## Examples
185+
186+
### Enriching evaluation results with MUTABLE
187+
188+
```python
189+
import wandb
190+
import numpy as np
191+
192+
run = wandb.init(project="mutable-logging")
193+
194+
# Step 1: Log initial predictions
195+
table = wandb.Table(columns=["input", "label", "prediction"], log_mode="MUTABLE")
196+
inputs, labels = load_eval_data()
197+
raw_preds = model.predict(inputs)
198+
199+
for inp, label, pred in zip(inputs, labels, raw_preds):
200+
table.add_data(inp, label, pred)
201+
202+
run.log({"eval_table": table}) # Log raw predictions
203+
204+
# Step 2: Add confidence scores (e.g. max softmax)
205+
confidences = np.max(raw_preds, axis=1)
206+
table.add_column("confidence", confidences)
207+
run.log({"eval_table": table}) # Add confidence info
208+
209+
# Step 3: Add post-processed predictions
210+
# (e.g., thresholded or smoothed outputs)
211+
post_preds = (confidences > 0.7).astype(int)
212+
table.add_column("final_prediction", post_preds)
213+
run.log({"eval_table": table}) # Final
214+
215+
run.finish()
216+
```
217+
218+
### Resuming runs with INCREMENTAL tables
219+
220+
You can continue logging to an incremental table when resuming a run:
221+
222+
```python
223+
# Start or resume a run
224+
resumed_run = wandb.init(project="resume-incremental", id="your-run-id", resume="must")
225+
226+
# Create the incremental table; no need to populate with data from preivously logged table
227+
# Increments will be continue to be added to the Table artifact.
228+
table = wandb.Table(columns=["step", "metric"], log_mode="INCREMENTAL")
229+
230+
# Continue logging
231+
for step in range(resume_step, final_step):
232+
metric = compute_metric(step)
233+
table.add_data(step, metric)
234+
resumed_run.log({"metrics": table}, step=step)
235+
236+
resumed_run.finish()
237+
```
238+
239+
{{% alert %}}
240+
Increments are logged to a new table if you turn off summaries on a key used for the incremental table using `wandb.define_metric("<table_key>", summary="none")` or `wandb.define_metric("*", summary="none")`.
241+
{{% /alert %}}
242+
243+
244+
### Training with INCREMENTAL batch training
245+
246+
```python
247+
248+
run = wandb.init(project="batch-training-incremental")
249+
250+
# Create an incremental table
251+
table = wandb.Table(columns=["step", "input", "label", "prediction"], log_mode="INCREMENTAL")
252+
253+
# Simulated training loop
254+
for step in range(get_num_batches()):
255+
# Load batch data
256+
inputs, labels = get_training_batch(step)
257+
258+
# Train the model on this batch
259+
train_model_on_batch(inputs, labels)
260+
261+
# Run model inference
262+
predictions = predict_on_batch(inputs)
263+
264+
# Add data to the table
265+
for input_item, label, prediction in zip(inputs, labels, predictions):
266+
table.add_data(step, input_item, label, prediction)
267+
268+
# Log the current state of the table incrementally
269+
run.log({"training_table": table}, step=step)
270+
271+
run.finish()

0 commit comments

Comments
 (0)