Expose the FailureReport type?

[thomasjm]

In the current Hspec Config object there's a configFailureReport option, which causes Hspec to dump a FailureReport into a file. However, it seems to just use the Show instance of FailureReport, so the output is hard to work with given that FailureReport (the constructor and potential Read instance) isn't exposed anywhere.

This is especially needed since hspec-jenkins doesn't work with Hspec version 2, leaving limited ways to get detailed test failure information out for downstream CI stuff without writing a custom Formatter or something. (The Summary returned by hspecResult only has success/failure counts.)

[sol]

Hey! Thanks for opening this issue and sorry for the late reply!

Currently the FailureReport type is used for rerunning failed tests (see https://hspec.github.io/rerun.html).

Let's look at that type:

data FailureReport = FailureReport {
  failureReportSeed :: Integer
, failureReportMaxSuccess :: Int
, failureReportMaxSize :: Int
, failureReportMaxDiscardRatio :: Int
, failureReportPaths :: [Path]
} deriving (Eq, Show, Read)

The first four fields are related to QuickCheck. The thing that may be of interest to you is failureReportPaths. This is basically a list of test cases that failed. E.g. with two failing test cases

describe "foo" $ do
  it "bar" ...
  it "baz" ...

failureReportPaths would be

[
  (["foo"], "bar")
, (["foo"], "baz")
]

Would it help your use case if you had access to this information?

[thomasjm]

No worries, it seems I'm using all your packages haha.

Yes, the failureReportPaths is what I'm interested in.

[sol]

@thomasjm Looking at again, I'm not comfortable with exposing Test.Hspec.Core.FailureReport in it's current form. What you can do as a workaround for now is defining

data FailureReport = FailureReport {
  failureReportSeed :: Integer
, failureReportMaxSuccess :: Int
, failureReportMaxSize :: Int
, failureReportMaxDiscardRatio :: Int
, failureReportPaths :: [Path]
} deriving (Eq, Show, Read)

in your own code for now and use that to read the failure report.

Is this a workable solution for you for now?

[thomasjm]

Sure, that works! It's not super urgent.

One other suggestion: it would be nice to add some more comments to the Haddocks on the Config constructor, since there's some useful stuff there but it's not totally clear what it all does. For example, both configFailureReport and configOutputFile accept a FilePath but I had to use trial and error to figure out what they each did.