Skip to content

Commit 321589c

Browse files
mkumar73mkumar73
committed
docs: fix TabularDataset kwargs and invalid numerical_preprocessing value
1 parent 76285e4 commit 321589c

11 files changed

Lines changed: 203 additions & 203 deletions

File tree

‎docs/getting_started/faq.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,10 +73,10 @@ from deeptab.data import TabularDataset
7373
from torch.utils.data import DataLoader
7474

7575
dataset = TabularDataset(
76-
cat_feature_list=[...],
77-
num_feature_list=[...],
78-
embedding_feature_list=None,
79-
y=labels,
76+
cat_features_list=[...],
77+
num_features_list=[...],
78+
embeddings_list=None,
79+
labels=labels,
8080
)
8181

8282
dataloader = DataLoader(dataset, batch_size=128, shuffle=True)

‎docs/model_zoo/experimental/modernnca.md‎

Lines changed: 37 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,17 @@ ModernNCA revisits Neighborhood Component Analysis (NCA) with modern tabular dee
1717

1818
This makes ModernNCA useful when the target function is locally smooth in a representation space: rows with similar learned embeddings should have similar labels.
1919

20-
| Property | DeepTab ModernNCA |
21-
| -------- | ----------------- |
22-
| Inductive bias | Local similarity / soft nearest-neighbor prediction |
23-
| Prediction form | Weighted candidate labels |
24-
| Training mode | Candidate-aware via `train_with_candidates` |
25-
| Inference cost | Pairwise distance to candidate rows |
26-
| Best baseline comparisons | TabR, TabM, ResNet, MLP |
20+
| Property | DeepTab ModernNCA |
21+
| ------------------------- | --------------------------------------------------- |
22+
| Inductive bias | Local similarity / soft nearest-neighbor prediction |
23+
| Prediction form | Weighted candidate labels |
24+
| Training mode | Candidate-aware via `train_with_candidates` |
25+
| Inference cost | Pairwise distance to candidate rows |
26+
| Best baseline comparisons | TabR, TabM, ResNet, MLP |
2727

2828
## Architectural Details
2929

30-
For a query row \(x_i\) and candidate rows \(\{x_j, y_j\}\), ModernNCA learns an encoder \(\phi_\theta\):
30+
For a query row \(x*i\) and candidate rows \(\{x_j, y_j\}\), ModernNCA learns an encoder \(\phi*\theta\):
3131

3232
```text
3333
raw features
@@ -44,11 +44,11 @@ embedding z = phi(x)
4444
Distances are converted to candidate weights:
4545

4646
\[
47-
d_{ij} = \frac{\|\phi_\theta(x_i) - \phi_\theta(x_j)\|_2}{T}
47+
d*{ij} = \frac{\|\phi*\theta(x*i) - \phi*\theta(x_j)\|\_2}{T}
4848
\]
4949

5050
\[
51-
w_{ij} = \mathrm{softmax}_j(-d_{ij})
51+
w*{ij} = \mathrm{softmax}\_j(-d*{ij})
5252
\]
5353

5454
For regression, the output is the weighted average of candidate targets. For classification, candidate labels are one-hot encoded and the weighted class probabilities are log-transformed before loss computation.
@@ -59,28 +59,28 @@ During training, DeepTab concatenates the current batch with a sampled subset of
5959

6060
The implementation lives in `deeptab/architectures/experimental/modern_nca.py`.
6161

62-
| Component | Implementation | Role |
63-
| --------- | -------------- | ---- |
64-
| Optional feature embedding | `EmbeddingLayer` when `use_embeddings=True` | Converts raw columns into per-feature representations |
65-
| Encoder | `nn.Linear(input_dim, config.dim)` | Projects the flattened row into metric space |
66-
| Post-encoder | Repeated BatchNorm -> Linear -> ReLU -> Dropout -> Linear blocks | Adds nonlinear representation capacity |
67-
| Candidate weighting | `torch.cdist` + `softmax(-distance / temperature)` | Differentiable neighbor weighting |
68-
| Candidate prediction | Matrix multiply between weights and candidate labels | Produces regression values or class probabilities |
69-
| Fallback head | `MLPhead` in `forward` | Allows non-candidate forward compatibility |
62+
| Component | Implementation | Role |
63+
| -------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------- |
64+
| Optional feature embedding | `EmbeddingLayer` when `use_embeddings=True` | Converts raw columns into per-feature representations |
65+
| Encoder | `nn.Linear(input_dim, config.dim)` | Projects the flattened row into metric space |
66+
| Post-encoder | Repeated BatchNorm -> Linear -> ReLU -> Dropout -> Linear blocks | Adds nonlinear representation capacity |
67+
| Candidate weighting | `torch.cdist` + `softmax(-distance / temperature)` | Differentiable neighbor weighting |
68+
| Candidate prediction | Matrix multiply between weights and candidate labels | Produces regression values or class probabilities |
69+
| Fallback head | `MLPhead` in `forward` | Allows non-candidate forward compatibility |
7070

7171
## Configuration
7272

73-
| Parameter | Default | Practical Effect |
74-
| --------- | ------- | ---------------- |
75-
| `dim` | `128` | Metric-space dimension after the encoder |
76-
| `d_block` | `512` | Hidden width inside residual post-encoder blocks |
77-
| `n_blocks` | `4` | Number of post-encoder blocks |
78-
| `dropout` | `0.1` | Regularization inside post-encoder blocks |
79-
| `temperature` | `0.75` | Softmax sharpness for candidate weighting |
80-
| `sample_rate` | `0.5` | Fraction of candidate rows sampled during training |
81-
| `embedding_type` | `"plr"` | Default embedding type when embeddings are enabled |
82-
| `n_frequencies` | `75` | PLR frequency count |
83-
| `frequencies_init_scale` | `0.045` | PLR initialization scale |
73+
| Parameter | Default | Practical Effect |
74+
| ------------------------ | ------- | -------------------------------------------------- |
75+
| `dim` | `128` | Metric-space dimension after the encoder |
76+
| `d_block` | `512` | Hidden width inside residual post-encoder blocks |
77+
| `n_blocks` | `4` | Number of post-encoder blocks |
78+
| `dropout` | `0.1` | Regularization inside post-encoder blocks |
79+
| `temperature` | `0.75` | Softmax sharpness for candidate weighting |
80+
| `sample_rate` | `0.5` | Fraction of candidate rows sampled during training |
81+
| `embedding_type` | `"plr"` | Default embedding type when embeddings are enabled |
82+
| `n_frequencies` | `75` | PLR frequency count |
83+
| `frequencies_init_scale` | `0.045` | PLR initialization scale |
8484

8585
```python
8686
from deeptab.configs import ModernNCAConfig, PreprocessingConfig, TrainerConfig
@@ -103,19 +103,19 @@ model = ModernNCAClassifier(
103103

104104
## Practical Guide
105105

106-
| Dataset Condition | Recommendation |
107-
| ----------------- | -------------- |
108-
| Small to medium data | ModernNCA is worth testing; candidate distance cost is manageable |
109-
| Very large candidate pool | Reduce `sample_rate`, use smaller batches, or prefer TabR/parametric models |
110-
| Noisy labels | Increase `temperature` or regularization; very sharp neighbor weights can overfit |
111-
| Strong local clusters | ModernNCA may be competitive with retrieval models |
112-
| Latency-sensitive inference | Prefer MLP/ResNet/TabM unless candidate search is acceptable |
106+
| Dataset Condition | Recommendation |
107+
| --------------------------- | --------------------------------------------------------------------------------- |
108+
| Small to medium data | ModernNCA is worth testing; candidate distance cost is manageable |
109+
| Very large candidate pool | Reduce `sample_rate`, use smaller batches, or prefer TabR/parametric models |
110+
| Noisy labels | Increase `temperature` or regularization; very sharp neighbor weights can overfit |
111+
| Strong local clusters | ModernNCA may be competitive with retrieval models |
112+
| Latency-sensitive inference | Prefer MLP/ResNet/TabM unless candidate search is acceptable |
113113

114114
Suggested search space:
115115

116116
```python
117117
param_grid = {
118-
"preprocessing_config__numerical_preprocessing": ["standard", "quantile", "ple"],
118+
"preprocessing_config__numerical_preprocessing": ["standardization", "quantile", "ple"],
119119
"model_config__dim": [64, 128, 256],
120120
"model_config__n_blocks": [2, 4, 6],
121121
"model_config__d_block": [256, 512],

‎docs/model_zoo/experimental/tangos.md‎

Lines changed: 40 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,13 @@ The research hypothesis is that tabular MLPs generalize better when hidden units
1515
- specialize on a sparse subset of input features, and
1616
- avoid learning highly overlapping feature attributions.
1717

18-
| Property | DeepTab Tangos |
19-
| -------- | -------------- |
20-
| Base architecture | MLP |
21-
| Additional mechanism | Jacobian-based specialization and orthogonalization penalty |
22-
| Training hook | `penalty_forward` |
23-
| Main cost driver | `torch.func.jacrev` / Jacobian computation |
24-
| Best baseline comparisons | MLP, ResNet, TabM |
18+
| Property | DeepTab Tangos |
19+
| ------------------------- | ----------------------------------------------------------- |
20+
| Base architecture | MLP |
21+
| Additional mechanism | Jacobian-based specialization and orthogonalization penalty |
22+
| Training hook | `penalty_forward` |
23+
| Main cost driver | `torch.func.jacrev` / Jacobian computation |
24+
| Best baseline comparisons | MLP, ResNet, TabM |
2525

2626
## Architectural Details
2727

@@ -42,7 +42,7 @@ Linear output head
4242
During training, Tangos computes a representation Jacobian:
4343

4444
\[
45-
J_{h,x} = \frac{\partial h(x)}{\partial x}
45+
J\_{h,x} = \frac{\partial h(x)}{\partial x}
4646
\]
4747

4848
where \(h(x)\) is the representation before the final output head. The model builds latent-unit attribution vectors from this Jacobian and adds:
@@ -53,38 +53,38 @@ where \(h(x)\) is the representation before the final output head. The model bui
5353
The training loss is:
5454

5555
\[
56-
\mathcal{L}_{total} = \mathcal{L}_{task} + \lambda_1 \mathcal{L}_{spec} + \lambda_2 \mathcal{L}_{orth}
56+
\mathcal{L}_{total} = \mathcal{L}_{task} + \lambda*1 \mathcal{L}*{spec} + \lambda*2 \mathcal{L}*{orth}
5757
\]
5858

5959
## Main Building Blocks
6060

6161
The implementation lives in `deeptab/architectures/experimental/tangos.py`.
6262

63-
| Component | Implementation | Role |
64-
| --------- | -------------- | ---- |
65-
| Dense body | `nn.ModuleList` of linear, normalization, activation, dropout layers | Learns tabular representation |
66-
| Optional GLU | `nn.GLU()` when `use_glu=True` | Gated dense transformations |
67-
| Optional skip connections | Shape-matched residual additions | Stabilizes deeper MLPs |
68-
| Representation function | `repr_forward` | Hidden representation used for Jacobian attribution |
69-
| Jacobian computation | `torch.func.vmap(torch.func.jacrev(...))` | Computes per-sample hidden-unit attributions |
70-
| Specialization loss | L1 norm of attribution tensor | Encourages sparse feature usage |
71-
| Orthogonality loss | Cosine similarity between neuron attributions | Encourages diverse hidden units |
72-
| Output head | `nn.Linear(last_hidden, num_classes)` | Task prediction |
63+
| Component | Implementation | Role |
64+
| ------------------------- | -------------------------------------------------------------------- | --------------------------------------------------- |
65+
| Dense body | `nn.ModuleList` of linear, normalization, activation, dropout layers | Learns tabular representation |
66+
| Optional GLU | `nn.GLU()` when `use_glu=True` | Gated dense transformations |
67+
| Optional skip connections | Shape-matched residual additions | Stabilizes deeper MLPs |
68+
| Representation function | `repr_forward` | Hidden representation used for Jacobian attribution |
69+
| Jacobian computation | `torch.func.vmap(torch.func.jacrev(...))` | Computes per-sample hidden-unit attributions |
70+
| Specialization loss | L1 norm of attribution tensor | Encourages sparse feature usage |
71+
| Orthogonality loss | Cosine similarity between neuron attributions | Encourages diverse hidden units |
72+
| Output head | `nn.Linear(last_hidden, num_classes)` | Task prediction |
7373

7474
## Configuration
7575

76-
| Parameter | Default | Practical Effect |
77-
| --------- | ------- | ---------------- |
78-
| `layer_sizes` | `[256, 128, 32]` | Width/depth of the MLP body |
79-
| `dropout` | `0.2` | Standard dropout regularization |
80-
| `activation` | `nn.ReLU()` | Hidden activation |
81-
| `use_glu` | `False` | Enables gated linear units |
82-
| `skip_connections` | `False` | Adds residual connections when shapes match |
83-
| `batch_norm` | inherited default `False` | Optional batch normalization |
84-
| `layer_norm` | inherited default `False` | Optional layer normalization |
85-
| `lamda1` | `0.5` | Weight for specialization penalty |
86-
| `lamda2` | `0.1` | Weight for orthogonality penalty |
87-
| `subsample` | `0.5` | Fraction used for regularization pair sampling |
76+
| Parameter | Default | Practical Effect |
77+
| ------------------ | ------------------------- | ---------------------------------------------- |
78+
| `layer_sizes` | `[256, 128, 32]` | Width/depth of the MLP body |
79+
| `dropout` | `0.2` | Standard dropout regularization |
80+
| `activation` | `nn.ReLU()` | Hidden activation |
81+
| `use_glu` | `False` | Enables gated linear units |
82+
| `skip_connections` | `False` | Adds residual connections when shapes match |
83+
| `batch_norm` | inherited default `False` | Optional batch normalization |
84+
| `layer_norm` | inherited default `False` | Optional layer normalization |
85+
| `lamda1` | `0.5` | Weight for specialization penalty |
86+
| `lamda2` | `0.1` | Weight for orthogonality penalty |
87+
| `subsample` | `0.5` | Fraction used for regularization pair sampling |
8888

8989
```python
9090
from deeptab.configs import PreprocessingConfig, TangosConfig, TrainerConfig
@@ -98,27 +98,27 @@ model = TangosRegressor(
9898
lamda2=0.1,
9999
subsample=0.5,
100100
),
101-
preprocessing_config=PreprocessingConfig(numerical_preprocessing="standard"),
101+
preprocessing_config=PreprocessingConfig(numerical_preprocessing="standardization"),
102102
trainer_config=TrainerConfig(lr=1e-3, batch_size=128, max_epochs=100),
103103
random_state=101,
104104
)
105105
```
106106

107107
## Practical Guide
108108

109-
| Dataset Condition | Recommendation |
110-
| ----------------- | -------------- |
111-
| Small or noisy data | Try Tangos against MLP/ResNet; the regularizer may help |
112-
| Very high feature count | Watch Jacobian memory and runtime |
113-
| Large batch sizes | Reduce batch size if Jacobian computation is slow or memory-heavy |
114-
| Need fast training | Prefer MLP, ResNet, or TabM |
115-
| Want attribution diversity analysis | Tangos is a useful research model |
109+
| Dataset Condition | Recommendation |
110+
| ----------------------------------- | ----------------------------------------------------------------- |
111+
| Small or noisy data | Try Tangos against MLP/ResNet; the regularizer may help |
112+
| Very high feature count | Watch Jacobian memory and runtime |
113+
| Large batch sizes | Reduce batch size if Jacobian computation is slow or memory-heavy |
114+
| Need fast training | Prefer MLP, ResNet, or TabM |
115+
| Want attribution diversity analysis | Tangos is a useful research model |
116116

117117
Suggested search space:
118118

119119
```python
120120
param_grid = {
121-
"preprocessing_config__numerical_preprocessing": ["standard", "quantile"],
121+
"preprocessing_config__numerical_preprocessing": ["standardization", "quantile"],
122122
"model_config__layer_sizes": [[128, 64], [256, 128, 32], [512, 256, 128]],
123123
"model_config__dropout": [0.0, 0.1, 0.2, 0.3],
124124
"model_config__lamda1": [0.1, 0.5, 1.0],

0 commit comments

Comments
 (0)