logs0:old

• Training w/o USERNAME token
  • replacing @username with USERNAME token caused high frequency of appearence of USERNAME. The result was poor.
  • So simply replacing it with "".
  • commit
• Make Attention work
  • Now beam/greedy can't work together.
• Testing Attention if it coverges with learning rate
  • Adam 0.05 and age decay 0.99 didn't work, perplexity stays around 50000-100000
  • Adam 0.5, didn't work
  • SGD with learning rate = 0.5 paramter no decay, worked very well.
    • perplexity goes down to around 80
    • おはよう -> おはようございます
    • commit
    • 'num_layers': 3, 'vocab_size': 50000, 'embedding_size': 1024
  • SGD 0.1 with conversation based data
• DONE Observation: See many dull responses, in normal seq2seq. So we try RL method.
  • Train normal seq2seq with larger vocab 50000 wasn't enough
  • Record how many dull responses does it have
  • Define dull responses list.
  • Check code is working
    • RL destination
    • learning rate etc.
    • It seems working
• DONE One more RL to verifiy it's working
• DONE Try beam search response to see if it's more diverse than befor
  • YES
• make the bot online again