fix(networks): only instantiate cross-attention layers when with_cross_attention=True

fix(networks): only instantiate cross-attention layers when with_cross_attention=True #22197