[Bug]: AzureStorageProvider Performance is Slow on Large Associated Storage Accounts

[mitchelsellers]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

When uploading, deleting, moving, or performing other actions using an AzureFolderProvider mapped file the performance of the operation degrades as the particular mapping grows and adds more files.,

Steps to reproduce?

1. Configure a Folder Mapping with an AzureFolder (Idealy a clean setup)
2. Upload a file, notice fast performance
3. Update the folder in Azure/DNN to have many files > 1000 or so and it seems to be visible
4. Upload a file againIdeally

Current Behavior

When completing an upload or other operation, you will find that the upload zooms to completed on the UI, but then it takes many seconds, up to 30-40 at times to refresh the listing and truly complete the operation.being completed on the UI, but then it takes many seconds, up to 30-40 at times,

Expected Behavior

This should be a quick operation and should not slow down with the addition of new files/folders.

Relevant log output

No response

Anything else?

In looking at this it seems to be directly related to the behaviors utilized with regards to cache management within the AzureFolderProvider itself.

After each operation you can see that it purges the cache -

Dnn.Platform/DNN Platform/Providers/FolderProviders/AzureFolderProvider/AzureFolderProvider.cs

Line 331 in 87bed09

this.ClearCache(folderMapping.FolderMappingID);

The action to re-fill the cache is here -

Dnn.Platform/DNN Platform/Providers/FolderProviders/AzureFolderProvider/AzureFolderProvider.cs

Line 241 in 87bed09

protected override IList<IRemoteStorageItem> GetObjectList(FolderMappingInfo folderMapping)

The issue is that since the mapping is per provider mapping, not per folder or otherwise, this requires a full enumeration of the Azure Blob storage account.

This has two net impacts to operators using Azure Storage Providers

• The performance issues as noted
• Additional billing in Azure as there is pricing for the number of reads etc. (Granted lower costs, but still a cost)

I think we need to look at better strategies for caching, or updating the cache for this provider.

Affected Versions

9.13.2 (latest release), 9.13.3 (alpha), 10.0.0 (alpha)

What browsers are you seeing the problem on?

Firefox, Chrome, Safari, Microsoft Edge

Code of Conduct

• I agree to follow this project's Code of Conduct

[valadas]

@mitchelsellers in addition to this, something calls SynchronizeFiles with a list that can contain a lot of folders (not files) and it throws a potentially huge amount of InvalidFileExtensionException in FolderManager.SynchronizeFiles

It gets entered though this call stack:

I setup a local environment here to troubleshoot this and what is taking the most time is getting the file LastModifiedDate for some reason. If you can join next co-coding session, happy to continue on that path...

[mitchelsellers]

Yes, I can try to join next week. I was in the air today when the session was going on