Proposal: allow exception marshallers via '[return: MarshalUsing]' on generated COM interfaces methods without '[PreserveSig]'

[Sergio0694]

Overview

This came up in microsoft/CsWinRT#1858. We'd like for the COM generator to support [return: MarshalUsing] to use custom marshallers for exceptions, so that CsWinRT could correctly allow using the necessary error propagation logic from generated COM interfaces. Currently, trying the code below doesn't do anything (but also it doesn't emit any warnings, it just silently does nothing).

Example

CsWinRT could expose this:

namespace WinRT;

[CustomMarshaller(typeof(Exception), MarshalMode.UnmanagedToManagedOut, typeof(ExceptionHelpersHResultMarshaller<>))]
public static class ExceptionHelpersHResultMarshaller<T>
    where T : unmanaged, INumber<T>
{
    public static T ConvertToUnmanaged(Exception e)
    {
        ExceptionHelpers.SetErrorInfo(e);

        return T.CreateTruncating(ExceptionHelpers.GetHRForException(e));
    }
}

And users could use it as follows:

[Guid("6234C2F7-9917-469F-BDB4-3E8C630598AF")]
[GeneratedComInterface(Options = ComInterfaceOptions.ManagedObjectWrapper)]
public partial interface IFoo
{
    [return: MarshalUsing(typeof(ExceptionHelpersHResultMarshaller<int>))]
    void Bar();
}

The COM generator would then use that in the catch block for the marshalling stub.

Risks

None that I can think of.

Alternatives

Developers can work around this by manually writing the entire vtable backend (eg. here). This is extremely clunky and error prone.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[jkotas]

What does this look like for methods with return value?

[Sergio0694]

My understanding is there would be no difference, because at the ABI level, the return is always an HRESULT anyway. So the generated ABI method will still always return an int __retVal local. This proposal is simply to allow overriding the default ExceptionAsHResultMarshaller<> type. Basically whenever that would be used by default by the COM generator, users should be able to indicate a custom equivalent type. Then the way the COM generator uses it would be identical to however ExceptionAsHResultMarshaller<> is already used today.

Eg. example codegen for:

[Guid("6234C2F7-9917-469F-BDB4-3E8C630598AF")]
[GeneratedComInterface(Options = ComInterfaceOptions.ManagedObjectWrapper)]
public partial interface IFoo
{
    double Bar();
}

You get:

file unsafe partial interface InterfaceImplementation
{
    [global::System.Runtime.InteropServices.UnmanagedCallersOnlyAttribute(CallConvs = new[] { typeof(global::System.Runtime.CompilerServices.CallConvMemberFunction) })]
    internal static int ABI_Bar(global::System.Runtime.InteropServices.ComWrappers.ComInterfaceDispatch* __this_native, double* __invokeRetValUnmanaged__param)
    {
        global::IFoo @this = default;
        ref double __invokeRetValUnmanaged = ref *__invokeRetValUnmanaged__param;
        double __invokeRetVal = default;
        int __retVal = default;
        try
        {
            // Unmarshal - Convert native data to managed data.
            @this = global::System.Runtime.InteropServices.ComWrappers.ComInterfaceDispatch.GetInstance<global::IFoo>(__this_native);
            __invokeRetVal = @this.Bar();
            // NotifyForSuccessfulInvoke - Keep alive any managed objects that need to stay alive across the call.
            __retVal = 0; // S_OK
            // Marshal - Convert managed data to native data.
            __invokeRetValUnmanaged = __invokeRetVal;
        }
        catch (global::System.Exception __exception)
        {
            __retVal = global::System.Runtime.InteropServices.Marshalling.ExceptionAsHResultMarshaller<int>.ConvertToUnmanaged(__exception);
        }

        return __retVal;
    }
}

We want to override that

__retVal = global::System.Runtime.InteropServices.Marshalling.ExceptionAsHResultMarshaller<int>.ConvertToUnmanaged(__exception);

[jkotas]

What about properties that return strings or other types that need marshalling? I can write this today:

public partial interface IFoo
{
    [return: MarshalUsing(typeof(BStrStringMarshaller))]
    string Bar();
}

If I want to override the exception marshaling too, do I specify two return marshalling directives? Is it always going to non-ambiguous how the two marshalling directives should be applied?

[Sergio0694]

I see what you mean. Right, that syntax was meant to be an example, but the real thing would likely need some other attribute targeting methods that would allow to indicate an exception marshaller. I just don't know which exact API surface it would require. If we want to brainstorm what an actual API proposal to bring to review would look like, I'd imagine a starting point could maybe be:

namespace System.Runtime.InteropServices.Marshalling
{
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
    public sealed class ExceptionMarshallerAttribute : Attribute
    {
        public ExceptionMarshallerAttribute(MarshalMode marshalMode, Type marshallerType);

        public MarshalMode MarshalMode { get; }
        public Type MarshallerType { get; }
    }
}

And then you'd use it as follows?

public partial interface IFoo
{
    [ExceptionMarshaller(MarshalMode.ManagedToUnmanagedIn, typeof(ExceptionHelpersHResultMarshaller<int>))]
    void Bar();
}

cc. @jkoritzinsky for thoughts, since you also mentioned we'd need a new API for this likely 🙂

EDIT: not entirely sure if we need MarshalMode, since it seems implied by the context? Maybe this is sufficient?

namespace System.Runtime.InteropServices.Marshalling
{
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
    public sealed class ExceptionMarshallerAttribute : Attribute
    {
        public ExceptionMarshallerAttribute(Type marshallerType);

        public Type MarshallerType { get; }
    }
}

And then you just do:

public partial interface IFoo
{
    [ExceptionMarshaller(typeof(ExceptionHelpersHResultMarshaller<int>))]
    void Bar();
}

[dongle-the-gadget]

I believe we should have a default exception handler for IErrorInfo, as well, since it looks like generated COM doesn't support that yet.