Skip to content

Commit 8f97685

Browse files
mkumar73mkumar73
committed
docs(model-zoo): update building block section
1 parent 6c1eaab commit 8f97685

10 files changed

Lines changed: 131 additions & 131 deletions

File tree

‎docs/model_zoo/stable/autoint.md‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,12 @@ feature tokens -> [LayerNorm -> MultiheadAttention -> residual -> Linear -> resi
2323

2424
## Main Building Blocks
2525

26-
| Component | DeepTab implementation | Role |
27-
| --- | --- | --- |
28-
| Tokenizer | `EmbeddingLayer` | Builds feature tokens. |
29-
| Interaction layer | `nn.MultiheadAttention` | Learns pairwise and higher-order token interactions. |
30-
| Residual projection | `nn.Linear(d_model, d_model)` | Updates each attended token. |
31-
| Output head | `nn.Linear(d_model * n_inputs, num_classes)` | Uses all token states for prediction. |
26+
| Component | DeepTab implementation | Role |
27+
| ------------------- | -------------------------------------------- | ---------------------------------------------------- |
28+
| Tokenizer | `EmbeddingLayer` | Builds feature tokens. |
29+
| Interaction layer | `nn.MultiheadAttention` | Learns pairwise and higher-order token interactions. |
30+
| Residual projection | `nn.Linear(d_model, d_model)` | Updates each attended token. |
31+
| Output head | `nn.Linear(d_model * n_inputs, num_classes)` | Uses all token states for prediction. |
3232

3333
## Implementation Notes
3434

@@ -57,12 +57,12 @@ model = AutoIntClassifier(
5757

5858
Key settings:
5959

60-
| Setting | Typical range | Effect |
61-
| --- | --- | --- |
62-
| `d_model` | `64` to `256` | Token width. |
63-
| `n_layers` | `2` to `6` | Number of interaction layers. |
64-
| `n_heads` | `4` to `8` | Attention heads; must divide `d_model`. |
65-
| `attn_dropout` | `0.0` to `0.3` | Attention regularization. |
60+
| Setting | Typical range | Effect |
61+
| ----------------------------- | ----------------- | ----------------------------------------------- |
62+
| `d_model` | `64` to `256` | Token width. |
63+
| `n_layers` | `2` to `6` | Number of interaction layers. |
64+
| `n_heads` | `4` to `8` | Attention heads; must divide `d_model`. |
65+
| `attn_dropout` | `0.0` to `0.3` | Attention regularization. |
6666
| `transformer_dim_feedforward` | Present in config | Not used by the current `AutoInt` architecture. |
6767

6868
## When To Use

‎docs/model_zoo/stable/enode.md‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,12 @@ feature tokens -> ENODEDenseBlock -> mean over feature axis -> Linear/ReLU/Dropo
2323

2424
## Main Building Blocks
2525

26-
| Component | DeepTab implementation | Role |
27-
| --- | --- | --- |
28-
| Tokenizer | `EmbeddingLayer` | Builds embedded feature tokens. |
29-
| Tree block | `ENODEDenseBlock` | Applies enhanced differentiable tree transformations. |
30-
| Aggregation | `x.mean(axis=1)` | Produces one row representation. |
31-
| Head | `nn.Linear -> ReLU -> Dropout -> nn.Linear` | Task output. |
26+
| Component | DeepTab implementation | Role |
27+
| ----------- | ------------------------------------------- | ----------------------------------------------------- |
28+
| Tokenizer | `EmbeddingLayer` | Builds embedded feature tokens. |
29+
| Tree block | `ENODEDenseBlock` | Applies enhanced differentiable tree transformations. |
30+
| Aggregation | `x.mean(axis=1)` | Produces one row representation. |
31+
| Head | `nn.Linear -> ReLU -> Dropout -> nn.Linear` | Task output. |
3232

3333
## Implementation Notes
3434

@@ -57,12 +57,12 @@ model = ENODERegressor(
5757

5858
Key settings:
5959

60-
| Setting | Typical range | Effect |
61-
| --- | --- | --- |
62-
| `d_model` | `4` to `32` | Embedded feature width. |
63-
| `num_layers` | `2` to `6` | Number of tree layers. |
64-
| `layer_dim` | `32` to `128` | Tree-layer width. |
65-
| `depth` | `4` to `8` | Soft decision depth. |
60+
| Setting | Typical range | Effect |
61+
| -------------- | -------------- | ------------------------------- |
62+
| `d_model` | `4` to `32` | Embedded feature width. |
63+
| `num_layers` | `2` to `6` | Number of tree layers. |
64+
| `layer_dim` | `32` to `128` | Tree-layer width. |
65+
| `depth` | `4` to `8` | Soft decision depth. |
6666
| `head_dropout` | `0.0` to `0.5` | Prediction-head regularization. |
6767

6868
## When To Use

‎docs/model_zoo/stable/fttransformer.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,13 @@ feature tokens -> TransformerEncoder x n_layers -> pooling -> optional norm -> M
2424

2525
## Main Building Blocks
2626

27-
| Component | DeepTab implementation | Role |
28-
| --- | --- | --- |
29-
| Tokenizer | `EmbeddingLayer` | Creates one vector per input feature. |
27+
| Component | DeepTab implementation | Role |
28+
| ------------- | ------------------------------- | ------------------------------------------------------ |
29+
| Tokenizer | `EmbeddingLayer` | Creates one vector per input feature. |
3030
| Encoder block | `CustomTransformerEncoderLayer` | Multi-head attention plus feed-forward transformation. |
31-
| Encoder stack | `nn.TransformerEncoder` | Repeats the block `n_layers` times. |
32-
| Pooling | `pooling_method`, `use_cls` | Reduces feature tokens to one representation. |
33-
| Head | `MLPhead` | Task-specific prediction head. |
31+
| Encoder stack | `nn.TransformerEncoder` | Repeats the block `n_layers` times. |
32+
| Pooling | `pooling_method`, `use_cls` | Reduces feature tokens to one representation. |
33+
| Head | `MLPhead` | Task-specific prediction head. |
3434

3535
## Implementation Notes
3636

@@ -61,13 +61,13 @@ model = FTTransformerClassifier(
6161

6262
Key settings:
6363

64-
| Setting | Typical range | Effect |
65-
| --- | --- | --- |
66-
| `d_model` | `64` to `256` | Token width and main capacity driver. |
67-
| `n_layers` | `2` to `6` | Transformer depth. |
68-
| `n_heads` | `4` to `8` | Attention heads; must divide `d_model`. |
69-
| `transformer_dim_feedforward` | `2x` to `4x d_model` | Feed-forward capacity. |
70-
| `pooling_method` | `"avg"` or `"cls"` | Sequence aggregation strategy. |
64+
| Setting | Typical range | Effect |
65+
| ----------------------------- | -------------------- | --------------------------------------- |
66+
| `d_model` | `64` to `256` | Token width and main capacity driver. |
67+
| `n_layers` | `2` to `6` | Transformer depth. |
68+
| `n_heads` | `4` to `8` | Attention heads; must divide `d_model`. |
69+
| `transformer_dim_feedforward` | `2x` to `4x d_model` | Feed-forward capacity. |
70+
| `pooling_method` | `"avg"` or `"cls"` | Sequence aggregation strategy. |
7171

7272
## When To Use
7373

‎docs/model_zoo/stable/mambattention.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,13 @@ feature tokens -> optional shuffle -> Mamba/Attention hybrid stack -> pooling ->
2424

2525
## Main Building Blocks
2626

27-
| Component | DeepTab implementation | Role |
28-
| --- | --- | --- |
29-
| Tokenizer | `EmbeddingLayer` | Builds one token per input feature. |
30-
| Mamba blocks | `ResidualBlock` inside `MambAttn` | Local/selective state-space sequence processing. |
31-
| Attention blocks | `nn.MultiheadAttention` | Explicit global token mixing. |
32-
| Hybrid schedule | `n_mamba_per_attention`, `n_attention_layers`, `last_layer` | Controls where attention is inserted. |
33-
| Head | `MLPhead` | Final task prediction. |
27+
| Component | DeepTab implementation | Role |
28+
| ---------------- | ----------------------------------------------------------- | ------------------------------------------------ |
29+
| Tokenizer | `EmbeddingLayer` | Builds one token per input feature. |
30+
| Mamba blocks | `ResidualBlock` inside `MambAttn` | Local/selective state-space sequence processing. |
31+
| Attention blocks | `nn.MultiheadAttention` | Explicit global token mixing. |
32+
| Hybrid schedule | `n_mamba_per_attention`, `n_attention_layers`, `last_layer` | Controls where attention is inserted. |
33+
| Head | `MLPhead` | Final task prediction. |
3434

3535
## Implementation Notes
3636

@@ -61,13 +61,13 @@ model = MambAttentionClassifier(
6161

6262
Key settings:
6363

64-
| Setting | Typical range | Effect |
65-
| --- | --- | --- |
66-
| `n_layers` | `2` to `6` | Mamba-block budget. |
67-
| `n_attention_layers` | `1` to `3` | Number of explicit attention insertions. |
68-
| `n_mamba_per_attention` | `1` to `3` | Frequency of attention layers. |
69-
| `last_layer` | `"attn"` or `"mamba"` | Final mixing type. |
70-
| `attn_dropout` | `0.0` to `0.3` | Attention regularization. |
64+
| Setting | Typical range | Effect |
65+
| ----------------------- | --------------------- | ---------------------------------------- |
66+
| `n_layers` | `2` to `6` | Mamba-block budget. |
67+
| `n_attention_layers` | `1` to `3` | Number of explicit attention insertions. |
68+
| `n_mamba_per_attention` | `1` to `3` | Frequency of attention layers. |
69+
| `last_layer` | `"attn"` or `"mamba"` | Final mixing type. |
70+
| `attn_dropout` | `0.0` to `0.3` | Attention regularization. |
7171

7272
## When To Use
7373

‎docs/model_zoo/stable/mambular.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,12 @@ feature tokens -> optional shuffle -> Mamba/MambaOriginal -> pooling -> MLPhead
2424

2525
## Main Building Blocks
2626

27-
| Component | DeepTab implementation | Role |
28-
| --- | --- | --- |
29-
| Tokenizer | `EmbeddingLayer` | Converts columns to a token sequence. |
27+
| Component | DeepTab implementation | Role |
28+
| -------------- | -------------------------- | -------------------------------------------------- |
29+
| Tokenizer | `EmbeddingLayer` | Converts columns to a token sequence. |
3030
| Sequence block | `Mamba` or `MambaOriginal` | Applies selective state-space sequence processing. |
31-
| Pooling | `pooling_method` | Reduces tokens to a row representation. |
32-
| Head | `MLPhead` | Task-specific prediction. |
31+
| Pooling | `pooling_method` | Reduces tokens to a row representation. |
32+
| Head | `MLPhead` | Task-specific prediction. |
3333

3434
## Implementation Notes
3535

@@ -59,12 +59,12 @@ model = MambularClassifier(
5959

6060
Key settings:
6161

62-
| Setting | Typical range | Effect |
63-
| --- | --- | --- |
64-
| `d_model` | `32` to `128` | Token width. |
65-
| `n_layers` | `2` to `6` | Number of Mamba blocks. |
66-
| `d_state` | `64` to `256` | State-space memory size. |
67-
| `d_conv` | `2` to `8` | Local convolution width inside Mamba. |
62+
| Setting | Typical range | Effect |
63+
| --------------- | ----------------- | ---------------------------------------------------- |
64+
| `d_model` | `32` to `128` | Token width. |
65+
| `n_layers` | `2` to `6` | Number of Mamba blocks. |
66+
| `d_state` | `64` to `256` | State-space memory size. |
67+
| `d_conv` | `2` to `8` | Local convolution width inside Mamba. |
6868
| `bidirectional` | `False` or `True` | Whether to process feature order in both directions. |
6969

7070
## When To Use

‎docs/model_zoo/stable/node.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,13 @@ features -> optional embeddings -> DenseBlock(num_layers, layer_dim, depth, tree
2323

2424
## Main Building Blocks
2525

26-
| Component | DeepTab implementation | Role |
27-
| --- | --- | --- |
28-
| Input representation | raw concatenation or `EmbeddingLayer` | Builds the vector consumed by trees. |
29-
| Differentiable trees | `deeptab.nn.blocks.node.DenseBlock` | Stacks NODE-style tree layers. |
30-
| Tree depth | `depth` | Controls number of soft splits per tree. |
31-
| Layer width | `layer_dim` | Number of trees/features per dense layer. |
32-
| Head | `MLPhead` | Maps tree representation to task output. |
26+
| Component | DeepTab implementation | Role |
27+
| -------------------- | ------------------------------------- | ----------------------------------------- |
28+
| Input representation | raw concatenation or `EmbeddingLayer` | Builds the vector consumed by trees. |
29+
| Differentiable trees | `deeptab.nn.blocks.node.DenseBlock` | Stacks NODE-style tree layers. |
30+
| Tree depth | `depth` | Controls number of soft splits per tree. |
31+
| Layer width | `layer_dim` | Number of trees/features per dense layer. |
32+
| Head | `MLPhead` | Maps tree representation to task output. |
3333

3434
## Implementation Notes
3535

@@ -57,12 +57,12 @@ model = NODEClassifier(
5757

5858
Key settings:
5959

60-
| Setting | Typical range | Effect |
61-
| --- | --- | --- |
62-
| `num_layers` | `2` to `6` | Number of dense tree layers. |
63-
| `layer_dim` | `64` to `256` | Width of each tree layer. |
64-
| `depth` | `4` to `8` | Soft decision depth. |
65-
| `tree_dim` | `1` to `3` | Output dimension per tree. |
60+
| Setting | Typical range | Effect |
61+
| ------------------ | --------------- | ------------------------------- |
62+
| `num_layers` | `2` to `6` | Number of dense tree layers. |
63+
| `layer_dim` | `64` to `256` | Width of each tree layer. |
64+
| `depth` | `4` to `8` | Soft decision depth. |
65+
| `tree_dim` | `1` to `3` | Output dimension per tree. |
6666
| `head_layer_sizes` | `[]` to `[128]` | Extra prediction-head capacity. |
6767

6868
## When To Use

‎docs/model_zoo/stable/saint.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,13 @@ feature tokens -> RowColTransformer -> pooling -> optional norm -> MLPhead
2424

2525
## Main Building Blocks
2626

27-
| Component | DeepTab implementation | Role |
28-
| --- | --- | --- |
29-
| Tokenizer | `EmbeddingLayer` | Converts each input feature to a token. |
30-
| Column attention | `nn.MultiheadAttention` inside `RowColTransformer` | Models feature interactions within a row. |
31-
| Row attention | Flattened row representation inside `RowColTransformer` | Mixes sample-level context within a batch. |
32-
| Feed-forward blocks | LayerNorm + Linear + activation + dropout | Adds nonlinear token updates. |
33-
| Prediction head | `MLPhead` | Produces final outputs. |
27+
| Component | DeepTab implementation | Role |
28+
| ------------------- | ------------------------------------------------------- | ------------------------------------------ |
29+
| Tokenizer | `EmbeddingLayer` | Converts each input feature to a token. |
30+
| Column attention | `nn.MultiheadAttention` inside `RowColTransformer` | Models feature interactions within a row. |
31+
| Row attention | Flattened row representation inside `RowColTransformer` | Mixes sample-level context within a batch. |
32+
| Feed-forward blocks | LayerNorm + Linear + activation + dropout | Adds nonlinear token updates. |
33+
| Prediction head | `MLPhead` | Produces final outputs. |
3434

3535
## Implementation Notes
3636

@@ -62,13 +62,13 @@ model = SAINTClassifier(
6262

6363
Key settings:
6464

65-
| Setting | Typical range | Effect |
66-
| --- | --- | --- |
67-
| `d_model` | `64` to `192` | Token width. |
68-
| `n_layers` | `1` to `4` | Row/column attention depth. |
69-
| `n_heads` | `2` to `8` | Number of attention heads. |
70-
| `attn_dropout`, `ff_dropout` | `0.0` to `0.3` | Regularization. |
71-
| `pooling_method`, `use_cls` | `"cls"`/`True` or `"avg"`/`False` | Token aggregation behavior. |
65+
| Setting | Typical range | Effect |
66+
| ---------------------------- | --------------------------------- | --------------------------- |
67+
| `d_model` | `64` to `192` | Token width. |
68+
| `n_layers` | `1` to `4` | Row/column attention depth. |
69+
| `n_heads` | `2` to `8` | Number of attention heads. |
70+
| `attn_dropout`, `ff_dropout` | `0.0` to `0.3` | Regularization. |
71+
| `pooling_method`, `use_cls` | `"cls"`/`True` or `"avg"`/`False` | Token aggregation behavior. |
7272

7373
## When To Use
7474

‎docs/model_zoo/stable/tabm.md‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,11 @@ features -> optional embeddings -> BatchEnsemble MLP blocks -> ensemble output/h
2424

2525
## Main Building Blocks
2626

27-
| Component | DeepTab implementation | Role |
28-
| --- | --- | --- |
29-
| Feature path | `EmbeddingLayer` or raw concatenation | Builds model input. |
30-
| Ensemble layers | `LinearBatchEnsembleLayer` | Shared weight matrix with member-specific scaling. |
31-
| Final layer | `SNLinear` or `nn.Linear` | Produces per-member or averaged predictions. |
27+
| Component | DeepTab implementation | Role |
28+
| --------------- | ----------------------------------------- | ------------------------------------------------------ |
29+
| Feature path | `EmbeddingLayer` or raw concatenation | Builds model input. |
30+
| Ensemble layers | `LinearBatchEnsembleLayer` | Shared weight matrix with member-specific scaling. |
31+
| Final layer | `SNLinear` or `nn.Linear` | Produces per-member or averaged predictions. |
3232
| Ensemble output | `returns_ensemble=True` when not averaged | Lets the training wrapper handle ensemble predictions. |
3333

3434
## Implementation Notes
@@ -59,13 +59,13 @@ model = TabMClassifier(
5959

6060
Key settings:
6161

62-
| Setting | Typical range | Effect |
63-
| --- | --- | --- |
64-
| `ensemble_size` | `8` to `64` | Number of virtual ensemble members. |
65-
| `layer_sizes` | `[128, 128]` to `[512, 256, 128]` | Shared MLP capacity. |
66-
| `model_type` | `"mini"` or `"full"` | Amount of member-specific scaling. |
67-
| `average_ensembles` | `False` or `True` | Return per-member outputs or average internally. |
68-
| `scaling_init` | `"ones"`, `"random-signs"`, `"normal"` | Diversity initialization for scaling factors. |
62+
| Setting | Typical range | Effect |
63+
| ------------------- | -------------------------------------- | ------------------------------------------------ |
64+
| `ensemble_size` | `8` to `64` | Number of virtual ensemble members. |
65+
| `layer_sizes` | `[128, 128]` to `[512, 256, 128]` | Shared MLP capacity. |
66+
| `model_type` | `"mini"` or `"full"` | Amount of member-specific scaling. |
67+
| `average_ensembles` | `False` or `True` | Return per-member outputs or average internally. |
68+
| `scaling_init` | `"ones"`, `"random-signs"`, `"normal"` | Diversity initialization for scaling factors. |
6969

7070
## When To Use
7171

0 commit comments

Comments
 (0)