PackageManager Async methods occasionally never complete

[ekalchev]

The AddPackageAsync, RemovePackageAsync, and StagePackageAsync methods in the PackageManager class occasionally fail to complete, causing the application to be indefinitely stuck in a waiting state. This issue is related to the one reported in CsWinRT issue #1720, but we have identified additional problematic methods in PackageManager. I won't be surprised if all PackageManager async methods to suffer from this issue, so far all methods that we attempted to use are buggy.

How to Reproduce

Compile the provided code in Release mode. We haven't tried Debug but from our previous experience with CsWinRT issue #1720 I might guess this is only Release build issue.
Execute the code, which repeatedly calls the StagePackageAsync, RegisterPackageByFamilyNameAsync, and RemovePackageAsync methods in a loop.
Observe that the task returned by StagePackageAsync frequently fails to complete, although RemovePackageAsync is also occasionally affected. When this happens a timeout exception is thrown, because of the CircuitBreaker. If you remove the CircuitBreaker the loop will be stuck on the await forever.

Observations

The issue typically reproduces at least once in every 500 iterations. Some machines exhibiting the problem more frequently than others.
Attaching a debugger reveals a worker thread perpetually waiting for task completion.
Attempts to resolve this by synchronously calling the async methods using GetAwaiter().GetResult() were unsuccessful.
The problem occurs on both Windows 10 and Windows 11 machines across various versions of the microsoft.windows.sdk.net.ref NuGet package.

Code Sample

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0-windows10.0.17763.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <Platforms>x64</Platforms>
  </PropertyGroup>

</Project>

using System;
using Windows.Management.Deployment;

namespace TestPackageInstall
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            string timestamp = DateTime.Now.ToString("HH:mm:ss");
            int numberOfSuccessfullInstallation = 0;
            int numberOfFailedInstallationWithTimeout = 0;
            int numberOfFailedInstallationUnknown = 0;

            while (true)
            {
                try
                {
                    // Create an instance of PackageManager
                    PackageManager packageManager = new PackageManager();

                    // Get the current time as a string in the format HH:MM:SS
                    timestamp = DateTime.Now.ToString("HH:mm:ss");

                    // Print the log messages with the timestamp
                    Console.WriteLine($"[{timestamp}] Number of successful installations: {numberOfSuccessfullInstallation}");
                    Console.WriteLine($"[{timestamp}] Number of failed installations, because of timeout: {numberOfFailedInstallationWithTimeout}");
                    Console.WriteLine($"[{timestamp}] Number of failed installations: {numberOfFailedInstallationUnknown}");
                    Console.WriteLine($"[{timestamp}] Staging...");

                    // Define the URI for the package
                    var uri = new Uri("https://storage.googleapis.com/ms-apps-bucket-exp/update/com.mobisystems.windows.appx.mobipdf/10.0.57990/full/MobiPDF.Package_10.0.57990.0_x64.msix");

                    // Attempt to stage the package
                    CircuitBreaker circuitBreaker = new CircuitBreaker();
                    await circuitBreaker.ExecuteAsync(async () => await packageManager.StagePackageAsync(uri, null).AsTask().ConfigureAwait(false), TimeSpan.FromMinutes(15)).ConfigureAwait(false);

                    timestamp = DateTime.Now.ToString("HH:mm:ss");
                    Console.WriteLine($"[{timestamp}] Register...");

                        // Register the package by family name
                        await packageManager.RegisterPackageByFamilyNameAsync("MobiSystems.MobiPdf_bvgb55c3tfatp", null, DeploymentOptions.ForceTargetApplicationShutdown, packageManager.GetDefaultPackageVolume(), null)
                            .AsTask(new ConsoleDeploymentProgress("RegisterPackage"))
                            .ConfigureAwait(false);
                    

                    // Increment the success counter
                    numberOfSuccessfullInstallation++;

                    timestamp = DateTime.Now.ToString("HH:mm:ss");
                    Console.WriteLine($"[{timestamp}] Uninstall...");

                    // Remove the package
                    await packageManager.RemovePackageAsync("MobiSystems.MobiPdf_10.0.57990.0_x64__bvgb55c3tfatp")
                        .AsTask(new ConsoleDeploymentProgress("RemovePackage"))
                        .ConfigureAwait(false);
                }
                catch (TimeoutException)
                {
                    timestamp = DateTime.Now.ToString("HH:mm:ss");

                    Console.WriteLine($"*********************************************************");
                    Console.WriteLine($"*********************************************************");
                    Console.WriteLine($"*********************************************************");
                    Console.WriteLine($"*********************************************************");
                    Console.WriteLine($"*********************************************************");
                    Console.WriteLine($"*********************************************************");
                    numberOfFailedInstallationWithTimeout++;
                }
                catch (Exception ex)
                {
                    timestamp = DateTime.Now.ToString("HH:mm:ss");

                    Console.WriteLine($"[{timestamp}] Exception: {ex.Message}");
                    numberOfFailedInstallationUnknown++;
                }

                timestamp = DateTime.Now.ToString("HH:mm:ss");
                // Clear the console for the next iteration
                Console.WriteLine($"[{timestamp}] ===============================================================");
            }
        }

        public class ConsoleDeploymentProgress : IProgress<DeploymentProgress>
        {
            private readonly string operationType;

            public ConsoleDeploymentProgress(string operationType)
            {
                this.operationType = operationType;
            }

            public void Report(DeploymentProgress value)
            {
                // Get the current time as a string in the format HH:MM:SS
                string timestamp = DateTime.Now.ToString("HH:mm:ss");
                Console.WriteLine($"[{timestamp}] {operationType} progress: {value.percentage}");
            }
        }

        public class CircuitBreaker
        {
            public async Task<T> ExecuteAsync<T>(Func<Task<T>> asyncMethod, TimeSpan timeout)
            {
                using (var cts = new CancellationTokenSource())
                {
                    var task = asyncMethod();
                    var completedTask = await Task.WhenAny(task, Task.Delay(timeout, cts.Token)).ConfigureAwait(false);

                    if (completedTask == task)
                    {
                        // Cancel the delay task if the main task completes first
                        cts.Cancel();
                        return await task.ConfigureAwait(false); // Unwrap and return the result
                    }
                    else
                    {
                        throw new TimeoutException("The operation has timed out.");
                    }
                }
            }
        }
    }
}

Example call stack when the method freezes

[ekalchev]

We have identified additional details regarding this issue. By passing a CancellationToken to the affected methods, the freezing problem is resolved. Here is an example of how to implement this:

using(CancellationTokenSource cts = new CancellationTokenSource ())
{
await packageManager.StagePackageAsync(uri, null).AsTask(cts .Token)
}

Using this approach, we conducted thousands of package installations with AddPackageAsync and StagePackageAsync, and none of these operations froze, unlike when they were executed without a CancellationToken. We hope this information helps in addressing the issue or provides a viable workaround.