[WIP] Implements RNNT+MMI

[pkufool]

It runs normally in my self-constructed test case, not fully tested yet, though.

The sampled paths:

sampled_paths = torch.tensor([ [ [ 3, 5, 0, 4, 6, 0, 2, 1 ],
                                 [ 2, 0, 5, 4, 0, 6, 1, 2 ],
                                 [ 3, 5, 2, 0, 0, 1, 6, 4 ]],
                               [ [ 7, 0, 4, 0, 6, 0, 3, 0 ],
                                 [ 0, 7, 3, 0, 2, 0, 4, 5 ],
                                 [ 7, 0, 3, 4, 0, 1, 2, 0 ]]], dtype=torch.int32)

The corresponding lattice:

Note: There is an arc from state 2 to state 17 in the second lattice, because the last symbol of the second path of second sequence is sampled at frame 1, it is a simulation of reaching final frame.

[pkufool]

@danpovey Do you have any good idea to test this function, I can only think of constructing simple test cases.

[pkufool]

I only include the "predictor" head output in C++ part, the other two scores (i.e. hybrid output and lm_output) will add on python part, it would be easier to enable autograd for hybrid output.

[pkufool]

path_scores here will contain hybrid_output and detached lm_output. I include the path_scores here and enable antograd to path_scores.

[danpovey]

Yes, OK. Right, we treat those as differentiable, but the negated sampling_prob is treated as just a constant.

[pkufool]

If we use regular RNN-T, it is possible to generate too many symbols for a specific frame, and that might be chances to generate a lattice containing cycles, which is not expected. I am not sure whether we will encounter such a issue at the very beginning of training.

[danpovey]

Hm, a valid point. Yes, computing forward backward scores would not work correctly if there are cycles. One possibility would be to augment the state with a sub-frame, i.e. instead of (ctx, t) it becomes (ctx, t, sub_t) with sub_t = (0, 1, 2, ..). That would prevent cycles, although it might prevent a small number of paths from recombining that might otherwise be able to recombine.