[RFC/WIP] class Example arg a instead of an associated type

[parsonsmatt]

This PR is more of a discussion piece - I'd like to test it more thoroughly in an app before suggesting it for real integration, since it is a substantive API change with unforeseen consequenes.

MultiParamTypeClasses vs TypeFamilies

The current Example class is defined with an associated type family Arg a which determines the environment or context that you can use to test an a.

class Example a where
    type Arg a

This is used for a few purposes:

1. Function arguments to examples, provided with hooks
2. Running fancier monads than IO
3. Not requiring it msg \ () -> True for non-function examples

The it function handily shows how this is used:

it :: Example a => String -> a -> SpecWith (Arg a)

With the instance Example Bool where type Arg Bool = (), we can fill this in:

it :: String -> Bool -> SpecWith ()

Making it a function (instance Example (a -> Bool) where type Arg (a -> Bool) = a, we can see how it expands out to accepting a function:

it :: String -> (a -> Bool) -> SpecWith a

As a result, we can write an additional case, as a specialization of the prior:

it :: String -> (() -> Bool) -> SpecWith ()

I'm gonna claim that the type family is overly restrictive, and that it'd be better to have a second parameter. The big reason is that we can be polymorphic in the Arg type, and the type family forbids that. For example, if we're testing a Bool, we really don't depend on the env at all. So we should be able to test like this:

it :: String -> Bool -> SpecWith Double

Polymorphic arg

With this formulation:

class Example env a

Let's compare the instance implementations:

instance Example env Bool where

instance Example Bool where
    type Arg Bool = ()

With the associated type, we can write it in any environment and provide a Bool and it "just works." This, to some extent, means that we don't even need to run the function to provide the value - so a pure test can be run very quickly and bypass any before_ that aren't useful for it!

Testing a Monad

So let's say you've got a few different monads - YesodExample, AppT, and DB. YesodExample needs an AppEnv and a Middleware, AppT needs an AppEnv only, and DB only requires a SqlBackend. Note that an (AppEnv, Middleware) contains all of these things (AppEnv has a SqlBackend inside). So we can, with the type family approach, express this:

instance Example YesodExample where
    type Arg YesodExample = (AppEnv, Middleware)

instance Example AppT where
    type Arg AppT = AppEnv

instance Example DB where
    type Arg DB = SqlBackend

However, this approach means that you can't write DB and YesodExample in the same block- if you have a Big withYesodExample :: SpecWith (AppEnv, Middleware) -> Spec, then you need to decompose that manually with before_.

You can instead choose to just have the Arg for each of these be hte maximal requirement, but that's unfortunate - it means we can't have special faster tests for DB only.

With the multi param type class, we can use Has-style constraints to make this work nicely.

instance (Has AppEnv env, Has Middleware env) => Example env YesodExample where
 
instance (Has AppEnv env) => Example env AppT

instance (Has SqlBackend env) => Example env DB

Now these can be seamlessly combined.

Function Examples

The current hooks API allows you to provide values to test examples. These are accessed with a function instance:

before :: IO a -> SpecWith a -> Spec

instance Example (a -> Bool) where
    type Arg (a -> Bool) = a

With the multiparam type class, this is expressed as so:

instance Example a (a -> Bool)

However, this has bad type inference properties: a much nicer approach uses the type equality trick.

instance (env ~ a) => Example env (a -> Bool)

But a natural extension to this is to require subtyping instead of equality. So we can also write:

instance (Has a env) => Example env (a -> Bool)

Now we can test an a -> Bool in a SpecWith a, or SpecWith (a, b).

What would be especially interesting is if we could also generalize the delegation and have a single function instance. Consider:

instance (Has a env, Example env r) => Example env (a -> r)

Unfortunately, this doesn't play nicely with type inference - a polymorphic function now doesn't play nicely as an input without a bunch of type annotations. However, many functions that are polymorphic simply don't deal with their argument: it msg \_ -> do ... can now simply be it msg do ....

There's one further behavior change with this function instance: the action is run once for QuickCheck properties, instead of multiple times.

    context "when used with a QuickCheck property" $ do
      it "runs action before every check of the property" $ do
        (rec, retrieve) <- mkAppend
        evalSpec_ $ H.before (rec "before" >> return "value") $ do
          H.it "foo" $ \value -> property $ \(_ :: Int) -> rec value
        retrieve `shouldReturn` (take 200 . cycle) ["before", "value"]

This test is the only failing test, once types are specified.

We can even write this:

    context "can provide two arguments to a function" $ do
        it "ok" $ do
            evalSpec_ $ H.before (pure (1 :: Int)) $ do
                H.beforeWith (\i -> pure ('a', i)) $ do
                    H.it "no args" $ do
                        pure () :: IO ()
                    H.it "one arg" $ \i -> do
                        (1 :: Int) `shouldBe` i
                    H.it "two args" $ \a i -> do
                        ('a', (1 :: Int)) `shouldBe` (a, i)
                    H.it "tuple" $ \(a, i) -> do
                        ('a', (1 :: Int)) `shouldBe` (a, i)

And, with the function instance, we no longer need to worry about ever writing the boilerplate function instances again.

[sol]

Quickly:

1. Composability of hooks is a weak spot that deserves attention.
2. Support for Bool is a historic artifact. Support for IO (), QuickCheck and custom stacks, I think that's what we care about. If you strip everything regarding Bool from the proposal then it will be an easier sell (to me at least).
3. From what I understand, your approach will hurt type inference. In many situations this may not be a big issue as we either have top level type annotations for the spec; or before and friends will restrain the involved types sufficiently. Questions: How does this affect error messages? If you don't care about hooks at all, can you still write a spec of type Spec without a type annotation?
4. Just to be sure, are you aware that you can provide SpecHook.hs-files for parts of the spec tree? E.g. you could have test/Model/SpecHook.hs that creates a db connection for all items in that branch of the tree and test/App/SpecHook.hs that creates a complete app environment for items under that branch of the tree. This does not allow for the level of flexibility you are proposing, but works reasonably well in many practical situations.
5. I think for experimentation it could be neat to provide your proposal as an alternative surface syntax. How much code duplication would that require? We will need a new Example or Testable class. Can we reuse the existing Spec type?
6. For completeness, it's not a given that Hspec's lawless Example type class / polymorphic approach is the sweet spot for a testing framework in Haskell. Ignoring hooks, having it specialized to IO () (or some dedicated UnitTest type) and prop specialized to QuickCheck properties provides some merit (better error messages on type errors).