Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix assertSnapshot for Swift Testing tests. #916

Merged
merged 3 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,11 @@ jobs:
strategy:
matrix:
xcode:
- "14.3.1"
- 15.4
- '16.0'

name: macOS 13 (Xcode ${{ matrix.xcode }})
runs-on: macos-13
name: macOS
runs-on: macos-14
steps:
- uses: actions/checkout@v3
- name: Select Xcode ${{ matrix.xcode }}
Expand Down
9 changes: 5 additions & 4 deletions Sources/SnapshotTesting/AssertSnapshot.swift
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
import XCTest

#if canImport(Testing)
// NB: We are importing only the implementation of Testing because that framework is not available
// in Xcode UI test targets.
@_implementationOnly import Testing
import Testing
#endif

/// Enhances failure messages with a command line diff tool expression that can be copied and pasted
Expand Down Expand Up @@ -421,7 +419,10 @@ public func verifySnapshot<Value, Format>(

if !attachments.isEmpty {
#if !os(Linux) && !os(Windows)
if ProcessInfo.processInfo.environment.keys.contains("__XCODE_BUILT_PRODUCTS_DIR_PATHS") {
if
ProcessInfo.processInfo.environment.keys.contains("__XCODE_BUILT_PRODUCTS_DIR_PATHS"),
!isSwiftTesting
{
XCTContext.runActivity(named: "Attached Failure Diff") { activity in
attachments.forEach {
activity.add($0)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,21 +67,14 @@ class FeatureTests: XCTestCase {

This will override the `diffTool` and `record` properties for each test function.

Swift's new testing framework does not currently have a public API for this kind of customization.
There is an experimental feature, called `CustomExecutionTrait`, that does gives us this ability,
and the library provides such a trait called ``Testing/Trait/snapshots(diffTool:record:)``. It can
be attached to any `@Test` or `@Suite` to configure snapshot testing:
Swift's new testing framework also allows for this kind of configuration, both for a single test
and an entire test suite. This is done via what are known as "test traits":

```swift
@_spi(Experimental) import SnapshotTesting
import SnapshotTesting

@Suite(.snapshots(record: .all, diffTool: .ksdiff))
struct FeatureTests {
}
```

> Important: You must import SnapshotTesting with the `@_spi(Experimental)` attribute to get access
to this functionality because Swift Testing's own `CustomExecutionTrait` is hidden behind the same
SPI flag. This means this API is subject to change in the future, but hopefully Apple will
publicize this tool soon.
Original file line number Diff line number Diff line change
Expand Up @@ -151,16 +151,12 @@ in an XCTest context or a Swift Testing context, and will determine if it should
`Issue.record` to trigger a test failure.

For the most part you can write tests for Swift Testing exactly as you would for XCTest. However,
there is one major difference. Swift Testing does not (yet) have a substitute for `invokeTest`,
which we used alongside `withSnapshotTesting` to customize snapshotting for a full test class.

There is an experimental version of this tool in Swift Testing, called `CustomExecutionTrait`, and
this library provides such a trait called ``Testing/Trait/snapshots(diffTool:record:)``. It allows
you to customize snapshots for a `@Test` or `@Suite`, but to get access to it you must perform an
`@_spi(Experimental)` import of snapshot testing:
there is one major difference. In order to override a snapshot's
[configuration](<doc:SnapshotTestingConfiguration>) for a particular test or an entire suite you
must use what are known as "test traits":

```swift
@_spi(Experimental) import SnapshotTesting
import SnapshotTesting

@Suite(.snapshots(record: .all, diffTool: .ksdiff))
struct FeatureTests {
Expand All @@ -169,7 +165,4 @@ struct FeatureTests {
```

That will override the `diffTool` and `record` options for the entire `FeatureTests` suite.

> Important: As evident by the usage of `@_spi(Experimental)` this API is subject to change. As
soon as the Swift Testing library finalizes its API for `CustomExecutionTrait` we will update
the library accordingly and remove the `@_spi` annotation.
These traits can also be used on individual `@Test`s too.
4 changes: 1 addition & 3 deletions Sources/SnapshotTesting/Internal/RecordIssue.swift
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
import XCTest

#if canImport(Testing)
// NB: We are importing only the implementation of Testing because that framework is not available
// in Xcode UI test targets.
@_implementationOnly import Testing
import Testing
#endif

var isSwiftTesting: Bool {
Expand Down
6 changes: 1 addition & 5 deletions Sources/SnapshotTesting/SnapshotsTestTrait.swift
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
#if canImport(Testing)
// NB: We are importing only the implementation of Testing because that framework is not available
// in Xcode UI test targets.
@_implementationOnly import Testing
import Testing

@_spi(Experimental)
Comment on lines -4 to -6
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads up @JRR-OSU and @kondratk. This will affect you based on our discussions in #901.

We have had to remove the @_implementationOnly imports for Testing because we want to remove the @_spi restriction from these testing traits. This means that once again Snapshot Testing will not work from UI testing targets. Again, we feel this is a bug for Apple to deal with, as Testing does not work in UI test targets.

The only other fix that comes to mind for this would be to move everything but the code that uses Swift Testing into some kind of "SnapshotTestingCore" module, and then that could be imported into UI test targets when needed. We will discuss this more.

extension Trait where Self == _SnapshotsTestTrait {
/// Configure snapshot testing in a suite or test.
///
Expand Down Expand Up @@ -33,7 +30,6 @@
}

/// A type representing the configuration of snapshot testing.
@_spi(Experimental)
public struct _SnapshotsTestTrait: SuiteTrait, TestTrait {
public let isRecursive = true
let configuration: SnapshotTestingConfiguration
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import Testing
import Foundation
import InlineSnapshotTesting
@_spi(Experimental) import SnapshotTesting
import SnapshotTesting

@Suite(
.snapshots(
Expand All @@ -21,6 +21,28 @@
}
}

@Test func inlineSnapshotFailure() {
withKnownIssue {
assertInlineSnapshot(of: ["Hello", "World"], as: .dump) {
"""
▿ 2 elements
- "Hello"

"""
}
} matching: { issue in
issue.description == """
Issue recorded: Snapshot did not match. Difference: …

@@ −1,3 +1,4 @@
 ▿ 2 elements
  - "Hello"
+ - "World"

"""
}
}

@Test func inlineSnapshot_NamedTrailingClosure() {
assertInlineSnapshot(
of: ["Hello", "World"], as: .dump,
Expand Down
2 changes: 1 addition & 1 deletion Tests/SnapshotTestingTests/AssertSnapshotSwiftTests.swift
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#if canImport(Testing)
import Testing
import Foundation
@_spi(Experimental) import SnapshotTesting
import SnapshotTesting

@Suite(
.snapshots(
Expand Down
2 changes: 1 addition & 1 deletion Tests/SnapshotTestingTests/SnapshotsTraitTests.swift
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#if compiler(>=6) && canImport(Testing)
import Testing
@_spi(Experimental) @_spi(Internals) import SnapshotTesting
@_spi(Internals) import SnapshotTesting

struct SnapshotsTraitTests {
@Test(.snapshots(diffTool: "ksdiff"))
Expand Down
25 changes: 25 additions & 0 deletions Tests/SnapshotTestingTests/SwiftTestingTests.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#if compiler(>=6) && canImport(Testing)
import Testing
import SnapshotTesting

@Suite(.snapshots(diffTool: "ksdiff"))
struct SwiftTestingTests {
@Test func testSnapshot() {
assertSnapshot(of: ["Hello", "World"], as: .dump)
}

@Test func testSnapshotFailure() {
withKnownIssue {
assertSnapshot(of: ["Goodbye", "World"], as: .dump)
} matching: { issue in
issue.description.hasSuffix("""
@@ −1,4 +1,4 @@
 ▿ 2 elements
− - "Hello"
+ - "Goodbye"
  - "World"
""")
}
}
}
#endif
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
▿ 2 elements
- "Hello"
- "World"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
▿ 2 elements
- "Hello"
- "World"
Loading