Differences from proptest-state-machine

[hamiltop]

Cool project! Captures a pattern we've used internally and makes things a little more clear.

How does this differ from https://github.com/proptest-rs/proptest/tree/master/proptest-state-machine and would it make sense to try and unify them? Or maybe they are separate but it still might make sense to pull this functionality into the main proptest repo and docs.

[nickelization]

Hi @hamiltop! That is a good question, and there is definitely some overlap between the two projects. I haven't had time to dig in too deeply to proptest-state-machine so I may be missing some key differences, but currently the biggest one is probably that proptest-stateful is geared more toward testing async APIs. (It can certainly be used to test non-async code, but it depends on tokio and uses async-trait to specify async trait methods, and also includes extra callbacks for test case setup/teardown.)

On the flip side, the proptest-state-machine library includes the ability to attempt shrinking individual operations themselves after it finishes the initial shrinking phase of removing entire operations. But I'm not sure how useful this capability would be in practice, because in most stateful property tests I've written, it's pretty likely that any shrinking attempt on an individual op will cause a precondition to be violated, which means the actual opportunities for shrinking in this way are often pretty rare. I've found that only removing entire ops generally gives good enough results in practice, anyway. We have thought about trying to add this as an enhancement in a way that works with preconditions, though, and there's more on this topic in issue #5.

When we first started working on proptest-stateful I actually checked a bunch of places (including the proptest repo) to see if anything like this existed or was being planned, and I didn't find anything about proptest-state-machine or any other similar libraries. If it had existed last year, we might have built off of that instead, although again, I haven't evaluated it that closely, so I don't know what kind of features it might be missing that we would need.

I also originally thought about trying to merge this code upstream to proptest early on, but thought it made more sense to keep it as a separate crate. However, maybe that thinking was misguided since now there is similar code that's been integrated into the main proptest repo 😄. At some point I may look into merging functionality from proptest-stateful upstream, but we would need to rewrite all the existing test suites we've already written at ReadySet if we wanted to deprecate use of our own library, so it's probably not a high priority project for us. Hopefully in the short term the community will benefit from having both options available!

[matthew-russo]

Hey, one of the proptest maintainers here. Cool to see variations of this pattern in play! If you have any learnings that you think are useful either for proptest-state-machine or ways to improve proptest in general, feel free to let us know. We purposely released proptest-state-machine as a separate crate so we could start with 0.* versioning in case we needed to make some breaking changes as we learned more about how people were using the pattern. We're open to accepting patches if you have the time and desire otherwise suggestions are greatly appreciated.

[nickelization]

Hey, thanks @matthew-russo! Yeah we started with 0.* with our crate for the same reason.

I noticed in the docs that proptest-state-machine draws inspiration from Erlang's eqc_statem, which funny enough is also where I first encountered stateful property testing, and the same key differences also ended up in proptest-stateful:

• I'd love to implement support for symbolic variables at some point but it is much harder to do in a language with a static type system; I tried early on and gave up, but would love to come back to this at some point.
• I would similarly love to add a concurrent version of this at some point, but haven't spent any time pursuing it yet.
• I also found it's easier to just use asserts in the code to run the op than to have a separate op-specific postcondition callback :)

Anyway, I'm about to be out on vacation for the majority of the next week or so, but when I'm back I'll definitely take some time to dig into proptest-state-machine and see if I can come up with any PRs and/or suggestions based on what I learned writing proptest-stateful! I've really enjoyed using the proptest library in general and would love to give back what I can.