Compute transition scores on CPU

[benjaminking]

This PR moves the computation of transition scores in SilTranslationPipeline to the CPU. The scores form a large tuple whose size is num_generated_tokens × batch_size × num_beams × vocab_size that can occupy a lot of the GPU's RAM. The tradeoff is that inference now takes 1.5-2x longer, since we're using the CPU.

We should discuss further whether this is a good tradeoff to make.

---

This change is 

[benjaminking]

One upside to the performance hit that I just realized: the EITL team has already been running inference with a batch size of 8 instead of 16, so computing transition scores on the CPU with a batch size of 16 will have comparable performance to computing transition scores on the GPU with a batch size of 8. So, this fix is at least no slower than the status quo.

Also, it looks like we compute transition scores whether or not we're calculating confidence. We may want to change that so to avoid an unnecessary performance hit calculating something that we're just throwing away.

@benjaminking made 1 comment.
Reviewable status: 0 of 1 files reviewed, all discussions resolved.