bug: directory names with emoji such as `Notes 📝` won't backup the files #5144

agung2001 · 2024-04-16T02:06:45Z

I have searched open and closed issues for duplicates.
I have searched the forum for related topics.

Environment info

Duplicati version: 2.0.7.1_beta_2023-05-25
Operating system: Linux/Ubuntu - Run on container lscr.io/linuxserver/duplicati
Backend: Google Drive

Description

directory names with emoji such as Notes 📝 won't backup the files

Steps to reproduce

Create a source directory with name that include emoji such as Notes 📝
Set duplicate backup with Google Drive as destination
Run the backup manually to initiate the versioning process
ERROR : Try to restore files, the directory will be empty with no files on it

Actual result:
directory names with emoji such as Notes 📝 won't backup the files
Expected result:
directory names with emoji such as Notes 📝 not empty and backing up the files inside it

Screenshots

Debug log

            {
  "DeletedFiles": 0,
  "DeletedFolders": 0,
  "ModifiedFiles": 0,
  "ExaminedFiles": 7437,
  "OpenedFiles": 0,
  "AddedFiles": 0,
  "SizeOfModifiedFiles": 0,
  "SizeOfAddedFiles": 0,
  "SizeOfExaminedFiles": 1803112545,
  "SizeOfOpenedFiles": 0,
  "NotProcessedFiles": 0,
  "AddedFolders": 0,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 0,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": null,
  "VacuumResults": null,
  "DeleteResults": {
    "DeletedSetsActualLength": 0,
    "DeletedSets": [],
    "Dryrun": false,
    "MainOperation": "Delete",
    "CompactResults": null,
    "ParsedResult": "Success",
    "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
    "EndTime": "2024-04-16T02:01:13.793969Z",
    "BeginTime": "2024-04-16T02:01:12.231092Z",
    "Duration": "00:00:01.5628770",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 5,
      "BytesUploaded": 0,
      "BytesDownloaded": 52805719,
      "FilesUploaded": 0,
      "FilesDownloaded": 3,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 1237,
      "KnownFileSize": 1519609297,
      "LastBackupDate": "2024-04-16T01:50:14+00:00",
      "BackupListCount": 1171,
      "TotalQuotaSpace": 2199023255552,
      "FreeQuotaSpace": 1979805888116,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-04-16T02:00:00.004174Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20231020T110000Z.dlist.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-i7c888945087547a68ebd9f614b4957af.dindex.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-bc61d36fa650c40909b0dce825e7b1d57.dblock.zip.aes",
        "Value": []
      }
    ],
    "ParsedResult": "Success",
    "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
    "EndTime": "2024-04-16T02:01:54.221517Z",
    "BeginTime": "2024-04-16T02:01:30.43985Z",
    "Duration": "00:00:23.7816670",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 5,
      "BytesUploaded": 0,
      "BytesDownloaded": 52805719,
      "FilesUploaded": 0,
      "FilesDownloaded": 3,
      "FilesDeleted": 0,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 1237,
      "KnownFileSize": 1519609297,
      "LastBackupDate": "2024-04-16T01:50:14+00:00",
      "BackupListCount": 1171,
      "TotalQuotaSpace": 2199023255552,
      "FreeQuotaSpace": 1979805888116,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2024-04-16T02:00:00.004174Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Success",
  "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
  "EndTime": "2024-04-16T02:01:54.227463Z",
  "BeginTime": "2024-04-16T02:00:00.004167Z",
  "Duration": "00:01:54.2232960",
  "MessagesActualLength": 12,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2024-04-16 02:00:00 +00 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
    "2024-04-16 02:00:41 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-04-16 02:01:05 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (1.21 KB)",
    "2024-04-16 02:01:13 +00 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted",
    "2024-04-16 02:01:13 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-04-16 02:01:29 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (1.21 KB)",
    "2024-04-16 02:01:30 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20231020T110000Z.dlist.zip.aes (445.62 KB)",
    "2024-04-16 02:01:46 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20231020T110000Z.dlist.zip.aes (445.62 KB)",
    "2024-04-16 02:01:46 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i7c888945087547a68ebd9f614b4957af.dindex.zip.aes (4.28 KB)",
    "2024-04-16 02:01:47 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i7c888945087547a68ebd9f614b4957af.dindex.zip.aes (4.28 KB)",
    "2024-04-16 02:01:47 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bc61d36fa650c40909b0dce825e7b1d57.dblock.zip.aes (49.92 MB)",
    "2024-04-16 02:01:54 +00 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bc61d36fa650c40909b0dce825e7b1d57.dblock.zip.aes (49.92 MB)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 5,
    "BytesUploaded": 0,
    "BytesDownloaded": 52805719,
    "FilesUploaded": 0,
    "FilesDownloaded": 3,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 1237,
    "KnownFileSize": 1519609297,
    "LastBackupDate": "2024-04-16T01:50:14+00:00",
    "BackupListCount": 1171,
    "TotalQuotaSpace": 2199023255552,
    "FreeQuotaSpace": 1979805888116,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-04-16T02:00:00.004174Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

The text was updated successfully, but these errors were encountered:

ts678 · 2024-04-16T14:18:36Z

directory names with emoji such as Notes 📝 won't backup the files

Did you try a restore, e.g. checkmark the folder that won't expand, and use "Pick location" to an empty folder. It works for me.

From what I can see (testing Windows to local folder), it looks like a problem in GUI restore tree, and not in the actual backup.

agung2001 · 2024-04-17T00:02:59Z

@ts678 I did the step that you've mentioned, but it seems it's still not working :

Check the source folder

Select empty folder for the destination

Warning Log

have tested both on my Mac and Linux server
on both OS the issue persist

ts678 · 2024-04-17T00:16:08Z

I suppose I could try Linux, but at least Mac test is non-Docker I think? Your total file count isn't super huge. You could try

The FIND command in GUI Commandline to list your 7437 files by giving it a *. Or give it a partial path and end it with a *.
For me, what fails is if I give it exactly the folder in question, and end with a slash to indicate it's a folder. Ends with * works.

agung2001 · 2024-04-17T02:38:54Z

aaa I see, yup you're right. I can find all lists of files and directories that are backed up via FIND command

also can restore the directory with no problem now
but I can only do that when restoring via the parent directory, not the emoji directory

for example, if I have a directory structure like so :

- company
  - Notes 📝

I can only restore it from the parent directory company instead of directly from Notes 📝

thx, @ts678 your help is very much appreciated 😁👍

ts678 · 2024-04-18T14:34:48Z

Something's wrong here somewhere, so developer help will be needed, but I'm glad you found the files were in the backup.
I had been taking a peek in the database, and the packets going from browser to server, to conclude searching looked bad.
What worries me a bit is that an emoji is just a Unicode character, and so the problem might be broader than is noted here.

Jojo-1000 · 2024-05-11T18:49:08Z

I can reproduce this under both Windows and Linux. It is important to point out that this still backs up all the files (as you found out).

The bug is somewhere in parsing the query string for the file name which should be loaded. The UI sends a request with a URL encoded string, but the C# code does not decode it correctly:

UI: filter=%40C%3A%5Cduplicati%5Ctest%5CEmoji%F0%9F%92%8B%F0%9F%98%98%5C (decoded: @C:\duplicati\test\Emoji💋😘\)
C# raw: filter=%40C%3A%5CDaten%5CCS%5CProjects%5Cduplicati%5Ctest%5CEmoji%F0%9F%92%8B%F0%9F%98%98%5C
C# decoded: filter=@C:\duplicati\test\Emoji%8B�%98%5C

Obviously, there is no folder called Emoji%8B�%98%5C, so the result is empty. If I manually fix the string, the correct files are found. It seems that the %F0%9F sequence somehow confuses the parser. Specifically HttpUtillityHelper.UrlDecode from the third-party HttpServer library seems to be broken.

The normal Uri class can decode the sequence properly.

However, I believe the goal is to move away from the custom HttpServer (#5150), so this will be resolved in the future. Right now there is no easy way to fix this.

ts678 · 2024-05-13T16:02:37Z

resolved in the future

Initial .NET8 release prioritized backlog has that last, and "If possible". There are asks to exit Beta soon, but also be rock solid. Maintainer response in the former topic looks like shipping wins over perfecting, and I'm not sure if new web server will ship.

Regardless, some future will probably see this, assuming things keep moving. Thanks BTW for continuing to make PRs ready.
That's always another question of which ones get in, and when.

ts678 · 2024-05-25T14:44:46Z

Can't restore files/folders with special characters in name #2004 (which just went stale) is similar, but the specifics are different.

Jojo-1000 · 2024-05-25T15:38:16Z

While that is similar, the cause is different. In this case, it is purely a UI to backend communication problem. The issue you linked has the error from the underlying list method.

kenkendk · 2024-05-27T09:07:35Z

Specifically HttpUtillityHelper.UrlDecode from the third-party HttpServer library seems to be broken.

The normal Uri class can decode the sequence properly.

However, I believe the goal is to move away from the custom HttpServer (#5150), so this will be resolved in the future. Right now there is no easy way to fix this.

Yikes, we also have a custom Url function in there. Great detective work.

I think this makes it higher priority to get the Kestrel webserver ready. It is top of my list now.

ts678 added the bug label Apr 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug: directory names with emoji such as `Notes 📝` won't backup the files #5144

bug: directory names with emoji such as `Notes 📝` won't backup the files #5144

agung2001 commented Apr 16, 2024

ts678 commented Apr 16, 2024

agung2001 commented Apr 17, 2024

ts678 commented Apr 17, 2024

agung2001 commented Apr 17, 2024

ts678 commented Apr 18, 2024

Jojo-1000 commented May 11, 2024 •

edited

ts678 commented May 13, 2024

ts678 commented May 25, 2024

Jojo-1000 commented May 25, 2024

kenkendk commented May 27, 2024

bug: directory names with emoji such as Notes 📝 won't backup the files #5144

bug: directory names with emoji such as Notes 📝 won't backup the files #5144

Comments

agung2001 commented Apr 16, 2024

Environment info

Description

Steps to reproduce

Screenshots

Debug log

ts678 commented Apr 16, 2024

agung2001 commented Apr 17, 2024

ts678 commented Apr 17, 2024

agung2001 commented Apr 17, 2024

ts678 commented Apr 18, 2024

Jojo-1000 commented May 11, 2024 • edited

ts678 commented May 13, 2024

ts678 commented May 25, 2024

Jojo-1000 commented May 25, 2024

kenkendk commented May 27, 2024

bug: directory names with emoji such as `Notes 📝` won't backup the files #5144

bug: directory names with emoji such as `Notes 📝` won't backup the files #5144

Jojo-1000 commented May 11, 2024 •

edited