[Examples] Standardize Predictors for ARD

[wjmaddox]

This isn't a major issue, but examples should that data should be standardized for ARD (and most other kernels) best practices and numerical stability. Otherwise, zero gradients for lengthscales can appear with very high variance data (e.g. skillcraft).

# model definition
class GPRegressionModel(gpytorch.models.ExactGP):
        def __init__(self, train_x, train_y, likelihood):
            super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)
            self.mean_module = gpytorch.means.ConstantMean()
            self.covar_module = gpytorch.kernels.RBFKernel(ard_num_dims=train_x.size(1))
        def forward(self, x):
            mean_x = self.mean_module(x)
            covar_x = self.covar_module(x)
            return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# generate very unbalanced data
torch.random.manual_seed(1000)
train_x = torch.cat([torch.randn(30,1)*5000, torch.randn(30,1)*0.001], dim=1)
train_y = torch.randn(30)

# and standardize it
train_x_std = train_x / train_x.std(0)

Then attempt to compute log probs for both non-standardized and standardized predictors:

lh = gpytorch.likelihoods.GaussianLikelihood()
model = GPRegressionModel(train_x, train_y, lh)
mll = gpytorch.mlls.ExactMarginalLogLikelihood(lh, model)
loss = -mll( model(train_x), train_y)
loss.backward()
print(model.covar_module.raw_lengthscale.grad)
# returns tensor([[0., 0.]])

lh = gpytorch.likelihoods.GaussianLikelihood()
model = GPRegressionModel(train_x, train_y, lh)
mll = gpytorch.mlls.ExactMarginalLogLikelihood(lh, model)
loss = -mll( model(train_x_std), train_y)
loss.backward()
print(model.covar_module.raw_lengthscale.grad)
# returns tensor([[-0.0136,  0.0462]])

Noticed from attempting to use ARD on skillcraft dataset by modifying the fast pred var example

[r69shabh]

I've submitted a PR to address this issue: #2751

Root Cause

When using ARD kernels with input data that has very different scales across dimensions (e.g. the skillcraft dataset), the kernel matrix computation exp(-0.5 * dist² / lengthscale²) can numerically underflow to zero — resulting in zero gradients for lengthscale parameters during training.

Fix

Added documentation notes to the RBFKernel and MaternKernel docstrings recommending that users standardize input data when using ARD kernels.

The notes clarify:

• This primarily affects ARD kernels (ard_num_dims is not None)
• With high-variance data, the kernel matrix can underflow to zero
• Standardizing inputs (subtract mean, divide by std) ensures numerical stability and proper gradient flow during training

Example

# Without standardization — zero gradients
loss.backward()
print(model.covar_module.raw_lengthscale.grad)
# tensor([[0., 0.]])

# With standardization — proper gradients
train_x_std = train_x / train_x.std(0)
loss.backward()
print(model.covar_module.raw_lengthscale.grad)
# tensor([[-0.0136, 0.0462]])

Let me know if any adjustments are needed!