Nice work and interesting results. It may be better and easier to use Transformer-XL for generation (github.com/kimiyoung/tran…).