Default Sampling Parameters

[BritishTeapot]

A lot of chat UIs only offer a limited amount of adjustable sampling parameters (for example, they quite often don't offer min-p), if any at all. Moreover, the chat UIs don't know much about the model or what it's recommended sampling parameters are, so we end up with sub-optimal defaults and poor quality gens. The idea is to let the default parameters (those, that are used if sampler config is not specified in the request) to be adjusted at start. Something like sampler_config.json file that contains the default sampler config to be used if no parameters are specified in the request, and can be used as a option when starting kcpp (like --default_sampling_param sampler_config.json).

[jabberjabberjabber]

You can do that already with --gendefaults

[LostRuins]

yup @jabberjabberjabber is right. You can also add --gendefaultsoverwrite to overwrite what the client sends

[BritishTeapot]

Thank you, so so much.

Then there must be a different issue. Help message does not display the option.

❯ ./koboldcpp-v1.104 --help | grep "gendefaults"
                        [--sdgendefaults {"parameter":"value",...}]
  --sdgendefaults {"parameter":"value",...}

Nor is it present in the wiki at all.

[LostRuins]

Please update to the latest koboldcpp release

[BritishTeapot]

Also, what options are the default in the first place? The same ones as in the koboldai ui?

[jabberjabberjabber]

Also, what options are the default in the first place? The same ones as in the koboldai ui?

temperature = 0.75
top_k = 100
top_a = 0.0
top_p = 0.92
min_p = 0.0
typical_p = 1.0
tfs = 1.0
nsigma = 0.0
rep_pen = 1.0 (with adapter_obj fallback)
rep_pen_range = 320
rep_pen_slope = 1.0
presence_penalty = 0.0
mirostat = 0
mirostat_tau = 5.0
mirostat_eta = 0.1
dry_multiplier = 0.0
dry_base = 1.75
dry_allowed_length = 2
dry_penalty_last_n = 320
dry_sequence_breakers = []
xtc_threshold = 0.2
xtc_probability = 0
dynatemp_range = 0.0
dynatemp_exponent = 1.0
smoothing_factor = 0.0
smoothing_curve = 1.0
adaptive_target = -1.0
adaptive_decay = 0.9
sampler_order = [6, 0, 1, 3, 4, 2, 5]
seed = -1
stop_sequence = []
ban_eos_token = False
bypass_eos_token = False
grammar = ""
grammar_retain_state = False
trimstop = True
logit_biases = {}
banned_tokens / banned_strings = []
custom_token_bans = ""
render_special = False
tool_call_fix = False