Add extensions API

[sol]

This PR adds an extension API that can be used to extend Hspec with additional functionality without the need to modify hspec-core.

An example extension that addresses #338:

module Test.Hspec.CI (use, only) where

import System.Environment

import Test.Hspec.Core.Extension
import Test.Hspec.Core.Extension.Item qualified as Item
import Test.Hspec.Core.Extension.Spec qualified as Spec
import Test.Hspec.Core.Extension.Tree qualified as Tree
import Test.Hspec.Core.Extension.Config qualified as Config

-- |
-- Do not run the spec items of the given subtree by default; execute them only
-- when the environment variable @CI@ is set.
--
-- It is idiomatic to use this in combination with `>>>`:
--
-- @
-- import Control.Arrow ((>>>))
-- import Test.Hspec
-- import qualified Test.Hspec.CI as CI
--
-- spec :: Spec
-- spec = do
--   describe ".." $ do
--     it ".." >>> CI.only $ do
--       ..
-- @
only :: SpecWith a -> SpecWith a
only = Spec.mapItems setCiOnly

data CiOnly = CiOnly

setCiOnly :: Item a -> Item a
setCiOnly = Item.setAnnotation CiOnly

getCiOnly :: Item a -> Maybe CiOnly
getCiOnly = Item.getAnnotation

newtype CiFlag = CI Bool

setCiFlag :: Bool -> Config -> Config
setCiFlag = Config.setAnnotation . CI

getCiFlag :: Config -> CiFlag
getCiFlag = fromMaybe (CI False) . Config.getAnnotation

use :: SpecWith a
use = do
  runIO (lookupEnv "CI") >>= \ case
    Nothing -> pass
    Just _ -> modifyConfig (setCiFlag True)
  registerOption $ flag "ci" setCiFlag "include itmes that are marked as \"CI only\""
  registerTransformation applyCiOnly

applyCiOnly :: Config -> [SpecTree] -> [SpecTree]
applyCiOnly config = case getCiFlag config of
  CI True -> id
  CI False -> Tree.mapItems $ \ item -> case getCiOnly item of
    Nothing -> item
    Just CiOnly -> Item.pendingWith message item
  where
    message = unlines [
        "This item is marked as \"CI only\" and excluded by default."
      , "Use `--ci' to include this item."
      ]

[BebeSparkelSparkel]

I think it would be helpful to have a specific environment lookup function or even better to have flag, option, ... automatically do an environment lookup. Either should have the standard HSPEC_ prefix.

[sol]

I think it would be helpful to have a specific environment lookup function or even better to have flag, option, ... automatically do an environment lookup. Either should have the standard HSPEC_ prefix.

I think that should already be the case, according to the rules documented here: https://hspec.github.io/options.html#specifying-options-through-environment-variables

[sol]

The reason that the example specifically looks up the CI environment variable is because this environment variable is already set for GitHub actions, but setting HSPEC_CI=yes will work as well.

[BebeSparkelSparkel]

It would be nice if the Test.Hspec.Core.Extension.option interface did not take the type OptionSetter and instead have interface functions for the common cases.

[sol]

It would be nice if the Test.Hspec.Core.Extension.option interface did not take the type OptionSetter and instead have interface functions for the common cases.

How exactly would that look like?

Right now you can say things like

registerOption $ flag "ci" setCiFlag
    "include itmes that are marked as \"CI only\""

or

registerOption $ option "tags" (argument "TAGS" return addTagFilters)
    "TAGS can be a list of tag names."

How would you want to express this instead?

[BebeSparkelSparkel]

In my pr I used the below

  , option "timeout" (argument "N" (fmap Seconds . readMaybe) (setTimeout . Just)) "each test will run at most N seconds"
   , mkOptionNoArg "no-timeout" Nothing (setTimeout Nothing) "remove test time limits"

So, it would be useful to have

optionalArg :: Read a => String -> String -> (a -> Config -> Config) -> Option
optionNoArg :: String -> (Config -> Config) -> Option

these are more clear than trying to figure out which constructor to use with option.

option and argument could then be moved to GetOpt.Declarative.Types for more advanced uses.

[sol]

mkOptionNoArg "no-timeout" Nothing (setTimeout Nothing) "remove test time limits"

I think this is currently not covered by the API, I'll try to look into it tomorrow.

[BebeSparkelSparkel]

Just some thoughts

[BebeSparkelSparkel]

It seems that registerOption will be used every time flag and option are used. I know you want to use them as internal functions but it would be better for the extension writers to have them return SpecWith a instead of Option.

[sol]

It's now registerOptions that expects a list of Option values.

Keeping the Option type Might allow for reuse of Test.Hspec.Core.Extension.Option for #870. So let's not abandon it, at least not yet.

[BebeSparkelSparkel]

Ok sounds good.

[BebeSparkelSparkel]

OptionSetter makes this hard to use and it would be better just have interface functions for all the common cases like

flag :: String -> Maybe String -> (Bool -> Config -> Config) -> String -> Option
optionalArg :: Read a => String -> Maybe String -> String -> (a -> Config -> Config) -> String -> Option
optionNoArg :: String -> Maybe String -> (Config -> Config) -> String -> Option
argument :: Read a => String -> (a -> Config -> Config) -> String -> Option

these are more clear than trying to figure out which constructor to use with option.

option and argument could then be moved to GetOpt.Declarative.Types for more advanced uses.

[sol]

https://github.com/hspec/hspec/pull/868/files#diff-89c0bd2feff5876f008665eb349641509884e5274c5eac99342be545d21d4bc3

Is this along the lines of what you had in mind?

[sol]

I'm not 100% sold on it, as it's not as clear anymore, which string is the option name, and which string is the name for the parameter. But let's give this a try, provisionally.

[BebeSparkelSparkel]

You are right about the which string is which. Perhaps aliases would be helpful.

argument :: CliName -> CliArgName -> CliHelp -> (Bool -> Config -> Config) -> Option
type CliName = String
type CliArgName = String
type CliHelp = String

I rearranged the arguments because I like having functions at the end which allows multiline lambdas without parenthaseses and hanging arguments. This should be applied to all the option functions if you choose to use it.

[BebeSparkelSparkel]

It seems strange to me to have 2 Config types especially when the other is not exported.

Also, since this is exported it locks in a lot of options in core that could actually be extensions. For example smallcheck is being deprecated.

[sol]

I think it's possible to remove some of those options after #870. However, only from this Config type. The fields of the core Config can only be deprecated, not removed, for backwards compatibility.

That's also the reason why I duplicated the type, so that the extension API doesn't "know about" legacy stuff.

It probably makes sense to finish #870 before the extensions API lands on main.

That's the approach I'm taking for now:

• The provisional API is on Make Hspec extensible #894, I use this as a staging ground
• Support custom options for examples #870 will go on Make Hspec extensible #894 once it's done
• Any additional changes will be merged to Make Hspec extensible #894
• Once everything looks good Make Hspec extensible #894 will go on main

This PR is now against #894, with the options and documentation changes.

[BebeSparkelSparkel]

Thank your for outlining the dev plan for extensions. Is there anything you want me to specifcally look at?