|
@lmthang | |||||
|
Introducing #MeenaBot, a 2.6B-param open-domain chatbot with near-human quality. Remarkably, we show strong correlation between perplexity & humanlikeness!
Paper: arxiv.org/abs/2001.09977
Sample conversations: github.com/google-researc… twitter.com/GoogleAI/statu… pic.twitter.com/3xNSV4r4uB
|
||||||
|
||||||
|
Thang Luong
@lmthang
|
28. sij |
|
#MeenaBot is based on the Evolved Transformer (ET, an improved Transformer) & trained to minimize perplexity, the uncertainty of predicting the next word in a conversation. We built a novel "shallow-deep" seq2seq architecture: 1 ET block for encoder & 13 ET blocks for decoder. pic.twitter.com/Mv2d4Los3k
|
||
|
|
||
|
Thang Luong
@lmthang
|
28. sij |
|
We design a new human evaluation metric, Sensibleness & Specificity Average (SSA), which captures key elements of natural conversations. SSA is also shown to correlate with humanlikeness while being easier to measure. Human scores 86% SSA, #MeenaBot 79%, other best chatbots 56%. pic.twitter.com/I7NKl2b9Tl
|
||
|
|
||
|
Thang Luong
@lmthang
|
28. sij |
|
Implications from the #MeenaBot project:
1. Perplexity might be "the" automatic metric that the field's been looking for.
2. Bots trained on large-scale social conversations & pushed hard for low perplexity will be good.
3. Safety layer is needed for respectful conversations! pic.twitter.com/WHrcstcglt
|
||
|
|
||
|
M Akash Kumar
@akashkm99
|
28. sij |
|
Nice work! Will the models be publicly released?
|
||
|
|
||
|
Thang Luong
@lmthang
|
29. sij |
|
We are actively working on this, so stay tune for more updates!
|
||
|
|
||
|
Asa Cooper Stickland
@AsaCoopStick
|
29. sij |
|
Awesome! I would love an appendix or similar with the ppl and exact architecture of the variants you tried out (e.g. ET vs vanilla Transformer)
|
||
|
|
||
|
Thang Luong
@lmthang
|
29. sij |
|
We created that table but didn't include because we felt like the specific details here do not matter a lot. what matters is showing the strong correlation exists & the community can verify with their own settlings.
|
||
|
|
||
|
Rajat Monga
@rajatmonga
|
29. sij |
|
|
||
|
|
||
|
Thang Luong
@lmthang
|
29. sij |
|
Thanks Rajat!
|
||
|
|
||
|
Jonathan Fly 👾
@jonathanfly
|
29. sij |
|
>near-human quality pic.twitter.com/mCgEpl8a3f
|
||
|
|
||
|
Jonathan Fly 👾
@jonathanfly
|
29. sij |
|
This one is really impressive pic.twitter.com/DmwirSGZKz
|
||
|
|
||