Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating a test's status that has already been reported results in a new test counter #4088

Open
thomhurst opened this issue Nov 20, 2024 · 0 comments

Comments

@thomhurst
Copy link
Contributor

This may not be considered a bug, so close if it won't be fixed.

Describe the bug

If a test that has previously been reported, is reported again with a new status, it is counted as another test, and doesn't just update the existing one.

Steps To Reproduce

Test A > Report Success
Test A > Report Failure (maybe a hook that ran after it failed, and you need to attach that failure to something?)

// See https://aka.ms/new-console-template for more information

using Microsoft.Testing.Platform.Builder;
using Microsoft.Testing.Platform.Capabilities;
using Microsoft.Testing.Platform.Capabilities.TestFramework;
using Microsoft.Testing.Platform.Extensions.Messages;
using Microsoft.Testing.Platform.Extensions.TestFramework;
using Microsoft.Testing.Platform.TestHost;

var builder = await TestApplication.CreateBuilderAsync(args);
builder.RegisterTestFramework(_ => new TestFrameworkCapabilities(), (_, _) => new DummyAdapter());
var app = await builder.BuildAsync();
return await app.RunAsync();

internal class DummyAdapter : ITestFramework, IDataProducer
{
    public string Uid => nameof(DummyAdapter);

    public string Version => string.Empty;

    public string DisplayName => string.Empty;

    public string Description => string.Empty;

    public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)];

    public Task<CloseTestSessionResult> CloseTestSessionAsync(CloseTestSessionContext context) => Task.FromResult(new CloseTestSessionResult { IsSuccess = true });

    public Task<CreateTestSessionResult> CreateTestSessionAsync(CreateTestSessionContext context) => Task.FromResult(new CreateTestSessionResult { IsSuccess = true });

    public Task ExecuteRequestAsync(ExecuteRequestContext context)
    {
        context.MessageBus.PublishAsync(this, new TestNodeUpdateMessage(new SessionUid("1"), new TestNode
        {
            Uid = "2",
            DisplayName = "Blah",
            Properties = new PropertyBag(PassedTestNodeStateProperty.CachedInstance)
        }));
        
        context.MessageBus.PublishAsync(this, new TestNodeUpdateMessage(new SessionUid("1"), new TestNode
        {
            Uid = "2",
            DisplayName = "Blah",
            Properties = new PropertyBag(new TimeoutTestNodeStateProperty(new Exception()))
        }));
     
        context.Complete();
        
        return Task.CompletedTask;
    }

    public Task<bool> IsEnabledAsync() => Task.FromResult(true);
}

internal class Capabilities : ITestFrameworkCapabilities
{
    IReadOnlyCollection<ITestFrameworkCapability> ICapabilities<ITestFrameworkCapability>.Capabilities
        => Array.Empty<ITestFrameworkCapability>();
}

Expected behavior

Previously reported status with same test Uid is overwritten with new status

Actual behavior

Updated status is counted as a new test

Test run summary: Failed!
total: 2
failed: 1
succeeded: 1
skipped: 0
duration: 71ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant