|
@Zergylord | |||||
|
Fair point, but it's worth noting that there is a trade-off between # seeds, # baselines, and # and complexity of environment(s).
Personally, I'd prefer a method /w 3 seeds eval'd on 57 tasks with 5 baselines to one with 10 seeds eval'd on 1 task with 2 baselines. twitter.com/pyoudeyer/stat…
|
||||||
|
||||||
|
Roman Werpachowski
@RWerpachowski
|
30. sij |
|
Tasks are not a straightforward replacement for seeds because seends can be randomised, and hand-made tasks are not. Unless you're procedurally generating the tasks, in which case you effectively have more seeds.
|
||
|
|
||
|
Steven Hansen
@Zergylord
|
30. sij |
|
I agree, not a replacement -- but they do compete for computing resources.
And generally I've found that it's easier to overfit to a single task than it is to be misled by too few seeds.
Both are problematic, but without infinite compute hard decisions must be made.
|
||
|
|
||