Beam Search 4.Visit seq2seq implmentation

Goal

Understand how decoder works. Especially how it handles y(t) and y(t + 1). Do they come at once?

Step by step

• predict.py
  • token_ids (input to encoder)
  • run
  • returns outputs (output from decoder)
• What type is output_logits?
  • seq2seq.outputs
  • seq2seq.outputs is first return value of tf.contrib.legacy_seq2seq.model_with_buckets
  • it's seq2seq(encoder_inputs[:bucket[0]], decoder_inputs[:bucket[1]]) where seq2seq is passed into the model_with_buckets
  • which is tf.contrib.legacy_seq2seq.embedding_attention_seq2seq
  • which is embedding_attention_decoder
  • which is attention_decoder
  • First, we run the cell on a combination of the input and previous attention masks: cell_output, new_state = cell(linear(input, prev_attn), prev_state). Then, we calculate new attention masks: new_attn = softmax(V^T * tanh(W * attention_states + U * new_state)) and then we calculate the output: output = linear(cell_output, new_attn).
• Who initiate outputs?
  • model initializer
• When outputs are filled?
  • okay, so reading code is a bit difficult. Any documents can help?
• I think it's better to check much simpler decoder which is rnn_decoder
  • outputs is list of output, where output is cell(...), so
  • what is cell(...)
    • In BasicLSTMCell
    • def call(self, inputs, state): is the one!
    • return output (graph for calculate using c, h and etc) and state
• How outputs are filed?
  • So my understanding is that wen we sess.run with encoder_inputs and outputs placeholders. It tries to compute placeholders. The outputs placeholders has dependencies where p[1] depends on p[0] and p[2] on p[1] and so on.
  • Each y(t) corresponds to cell (whatever type it is)
  • So I think y(t) is just made by y(t - 1) and some states.
  • It might be a bit hard to hook when it happens? So as someone suggests do it in graph is better?