Issue/1127: kv replica when tp_size > num_kv_heads #4954

Job	Run time
Build (ubuntu-latest, debug)	15m 54s
Build (windows-latest, debug)	21m 16s
Build (ubuntu-latest, release)	16m 26s
Build (windows-latest, release)	19m 47s
	1h 13m 23s

Provide feedback