|
@jepsen_io | |||||
|
But still, some vendors do misrepresent the results of our work together, and this bugs me. I expected vendors to call each other out for this sort of thing, because they're more than willing to write take-downs over interpretation of benchmarks, but so far that hasn't happened.
|
||||||
|
||||||
|
Jepsen
@jepsen_io
|
18. ruj |
|
Again, YugaByte DB's Jepsen tests did not pass. They do not currently pass. Correctness issues we identified in our collaboration, mainly due to non-transactional schema changes, are still unaddressed. YugaByte and I have talked about this.
blog.yugabyte.com/announcing-yug… pic.twitter.com/lu8gwTq8re
|
||
|
|
||
|
Jepsen
@jepsen_io
|
18. ruj |
|
Because these problems involve schema changes (e.g. creating tables), they may not impact users frequently. YugaByte doesn't think they're relevant to the core transactional mechanism in YugaByte DB, which is why they're not discussing them when they say "Jepsen tests passed".
|
||
|
|
||
|
Jepsen
@jepsen_io
|
18. ruj |
|
There are other issues that we found in our work, like slowly spawning an ever-increasing number of backend worker processes which eventually consume all resources and kill the machine. This one's still open too. github.com/YugaByte/yugab…
|
||
|
|
||
|
Jepsen
@jepsen_io
|
18. ruj |
|
Crashes, unavailability, and performance problems aren't usually reported by Jepsen as "failing" results, because we're primarily concerned with checking safety, rather than liveness problems. It's hard to say how slow is too slow. We file and discuss these issues qualitatively.
|
||
|
|
||
|
Jepsen
@jepsen_io
|
18. ruj |
|
In more general terms, almost every database we tested with Jepsen fails, sometimes in dozens of ways, before its test suite passes. That's how we know Jepsen is *working*! The vendor headline that comes out of that process is usually "X passes Jepsen". pic.twitter.com/bJBT3MDw74
|
||
|
|
||
|
Jepsen
@jepsen_io
|
18. ruj |
|
Most vendors are telling the truth here: by the time we conclude our collaboration, the safety issues we found have usually been addressed, and the test suite often passes. Many vendors also follow up "passes" with a description of the issues we found, which I think is honest.
|
||
|
|
||
|
Jepsen
@jepsen_io
|
18. ruj |
|
I'm committed to giving everyone the most accurate, rigorous reporting on database correctness that I can, and I encourage vendors to do the same. Be open, honest, and nuanced in your writing. That honesty is good for users, and it builds trust in your team.
|
||
|
|
||