API Bug: OneDrive Business Shared Folders: Unable to sync shared folders, shared from outside organisation

[abraunegg]

Shared folders, shared with you from people outside of your 'organisation' are unable to be synced. This is due to the Microsoft Graph API not presenting these folders.

Shared folders that match this scenario, when you view 'Shared' via OneDrive online, will have a 'world' symbol as per below:

This can only be fixed once the Microsoft Graph API is fixed.

[abraunegg]

Opened API Issue: OneDrive/onedrive-api-docs#1301

[abraunegg]

Also commented here: https://github.com/microsoftgraph/microsoft-graph-docs-contrib/issues/7858

[mickaelleclercq]

The issue seem to remain.
I have folder shared from external organization that I can see in the browser, but onedrive --list-shared-folders cannot

[abraunegg]

@mickaelleclercq

The issue seem to remain.
I have folder shared from external organization that I can see in the browser, but onedrive --list-shared-folders cannot

This is a known issue, hence why I opened this issue ticket to tracking and visibility. This is also why it is listed as a known issue here: https://github.com/abraunegg/onedrive/blob/master/docs/BusinessSharedFolders.md#known-issues

The 'master' codebase does not have the required fix to handle external organisation shared folders, although there is a PR which contains a potential fix.

To assist with testing the current fix, please can you test the following PR which may resolve the issue, however in my testing the OneDrive API issue still exists - as in, whilst shared folders are now listed, accessing / syncing them fails as the OneDrive API does not authorise the access correctly. This could be, for my case, due to using a 'developer' Business account + shared folder from a commercial organisation. The API failure might be legitimate in this case.

Please test the following PR in your situation:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/969/head:pr969
git checkout pr969
./configure; make clean; make;

When running the PR, your version should be: onedrive v2.4.8-28-g2a93b4e or greater.

This is a work-in-progress (WIP) branch. It may contain other defects and needs extensive testing. Use with caution.

[mickaelleclercq]

After what you propose, I ended with the very last version:

$ onedrive --version onedrive v2.4.8-28-g2a93b4e

Now I have this error. With the previous version I had no results, so it there is improvement !

$onedrive --list-shared-folders
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...

Listing available OneDrive Business Shared Folders:
onedrive.OneDriveException@src/onedrive.d(1464): HTTP request returned status code 403 (Forbidden)
{
    "error": {
        "code": "Authorization_RequestDenied",
        "innerError": {
            "client-request-id": "a9cb5512-ea4d-40b3-9688-bf1b59bb635b",
            "date": "2020-12-11T21:11:13",
            "request-id": "a9cb5512-ea4d-40b3-9688-bf1b59bb635b"
        },
        "message": "Insufficient privileges to complete the operation."
    }
}
----------------
??:? void onedrive.OneDriveApi.checkHttpCode(ref const(std.json.JSONValue)) [0x564db2654678]
??:? std.json.JSONValue onedrive.OneDriveApi.get(const(char)[], bool) [0x564db2652b0b]
??:? std.json.JSONValue onedrive.OneDriveApi.getTenantID() [0x564db2650648]
??:? void sync.SyncEngine.listOneDriveBusinessSharedFolders() [0x564db2674e6f]
??:? _Dmain [0x564db26498d3]
std.net.curl.CurlException@std/net/curl.d(4402): A libcurl function was given a bad argument on handle null
----------------
??:? pure @safe void std.exception.bailOut!(std.net.curl.CurlException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x564db26e4086]
??:? pure @safe bool std.exception.enforce!(std.net.curl.CurlException).enforce!(bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x564db26e3ff7]
??:? void std.net.curl.Curl._check(int) [0x564db26b38cb]
??:? void std.net.curl.Curl.clear(etc.c.curl.CurlOption) [0x564db26b3c76]
??:? void std.net.curl.HTTP.clearRequestHeaders() [0x564db26b29ab]
??:? void onedrive.OneDriveApi.shutdown() [0x564db264f5ec]

[mickaelleclercq]

Note that onedrive --synchronize --dry-run works like a charm, so the privileges are working normally for the non-shared content

[abraunegg]

@mickaelleclercq
For this command onedrive --list-shared-folders - please provide a verbose debug log as per the support requirements: https://github.com/abraunegg/onedrive/wiki/Generate-debug-log-for-support

[mickaelleclercq]

There are a lot of personal information in it ! I can't send this here

[abraunegg]

@mickaelleclercq
Please rebuild the PR - made a tweak to add some additional error handling.

The new PR version is onedrive v2.4.8-29-g0e886ac

[mickaelleclercq]

Installed the new one

$ onedrive --version
onedrive v2.4.8-5-g9b44bb1

Here is the output or shared folders request:

$ onedrive --list-shared-folders
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...

Listing available OneDrive Business Shared Folders:

And the verbose:

[DEBUG] homePath: HOME environment variable set
[DEBUG] homePath: /root
[DEBUG] configDirBase: WARNING - no XDG_CONFIG_HOME environment variable set
[DEBUG] configDirBase: /root/.config
[DEBUG] configDirName: Configuring application to use default config path
Using 'user' Config Dir: /root/.config/onedrive
Using 'system' Config Dir: /etc/onedrive
[DEBUG] refreshTokenFilePath = /root/.config/onedrive/refresh_token
[DEBUG] deltaLinkFilePath = /root/.config/onedrive/delta_link
[DEBUG] databaseFilePath = /root/.config/onedrive/items.sqlite3
[DEBUG] databaseFilePathDryRun = /root/.config/onedrive/items-dryrun.sqlite3
[DEBUG] uploadStateFilePath = /root/.config/onedrive/resume_upload
[DEBUG] userConfigFilePath = /root/.config/onedrive/config
[DEBUG] syncListFilePath = /root/.config/onedrive/sync_list
[DEBUG] systemConfigFilePath = /etc/onedrive/config
[DEBUG] businessSharedFolderFilePath = /root/.config/onedrive/business_shared_folders
No user or system config file found, using application defaults
[DEBUG] sync_dir: Getting syncDir from config value sync_dir
[DEBUG] sync_dir: A '~' was found in configured sync_dir, automatically expanding as SHELL and USER environment variable is set
[DEBUG] syncDir: /root/OneDrive
[DEBUG] Testing network to ensure network connectivity to Microsoft OneDrive Service
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
[DEBUG] Configured authUrl:          https://login.microsoftonline.com/common/oauth2/v2.0/authorize
[DEBUG] Configured redirectUrl:      https://login.microsoftonline.com/common/oauth2/nativeclient
[DEBUG] Configured tokenUrl:         https://login.microsoftonline.com/common/oauth2/v2.0/token
[DEBUG] Configured driveUrl:         https://graph.microsoft.com/v1.0/me/drive
[DEBUG] Configured driveByIdUrl:     https://graph.microsoft.com/v1.0/drives/
[DEBUG] Configured sharedWithMeUrl:  https://graph.microsoft.com/v1.0/me/drive/sharedWithMe
[DEBUG] Configured itemByIdUrl:      https://graph.microsoft.com/v1.0/me/drive/items/
[DEBUG] Configured itemByPathUrl:    https://graph.microsoft.com/v1.0/me/drive/root:/
[DEBUG] Configured siteSearchUrl:    https://graph.microsoft.com/v1.0/sites?search
[DEBUG] Configured siteDriveUrl:     https://graph.microsoft.com/v1.0/sites/
[DEBUG] Downgrading all HTTP operations to HTTP/1.1 by default
[DEBUG] clientId    = d50ca740-c83f-4d1b-b616-12c519384f0c
[DEBUG] companyName = abraunegg
[DEBUG] appTitle    = OneDrive Client for Linux
Opening the item database ...
[DEBUG] Using database file: /root/.config/onedrive/items.sqlite3
[DEBUG] Configuring default new folder permissions as: 700
[DEBUG] Configuring default new file permissions as: 600
All operations will be performed in: /root/OneDrive
[DEBUG] Configuring skip_dir ...
[DEBUG] skip_dir: 
[DEBUG] Configuring skip_dir_strict_match ...
[DEBUG] skip_dir_strict_match: false
[DEBUG] Configuring skip_dotfiles ...
[DEBUG] skip_dotfiles: false
[DEBUG] Configuring skip_file ...
[DEBUG] skip_file: ~*|.~*|*.tmp
[DEBUG] Setting file permissions for: /root/.config/onedrive/refresh_token
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive/root
[DEBUG] OneDrive Account Details:     ***************
[DEBUG] OneDrive Account Root Details:  ****************
Application version: v2.4.8-5-g9b44bb1
Account Type: business
Default Drive ID: **********
Default Root ID: *******
Remaining Free Space: 967786905062
Fetching details for OneDrive Root
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive/root
[DEBUG] lastModifiedDateTime (OneDrive item): 2020-Dec-11 22:48:48Z
OneDrive Root exists in the database
Initializing the Synchronization Engine ...

Listing available OneDrive Business Shared Folders:
[DEBUG] Request URL = https://graph.microsoft.com/v1.0/me/drive/sharedWithMe
[DEBUG] onedrive.getSharedWithMe API Response: 

In the "onedrive.getSharedWithMe API Response: " I see all files that have been shared by other peoples in my organization. The folder shared by external organization does not appear.
But it is not your fault. I tried GET https://graph.microsoft.com/v1.0/me/drive/sharedWithMe?allowexternal=true and it is the same reponse...

[abraunegg]

@mickaelleclercq
It appears you are not running the PR version:

Application version: v2.4.8-5-g9b44bb1

When running the PR version you application version should be:

Application version: v2.4.8-29-g0e886ac

[mickaelleclercq]

Is there a way to retrieve it ?
I ran this:

rm -fr onedrive/
sudo -s
apt update
git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/969/head:pr969
git checkout pr969
source ~/dlang/dmd-2.094.2/activate
./configure; make clean; make;
deactivate

$onedrive --version
onedrive v2.4.8-5-g9b44bb1

[abraunegg]

Run the PR from the build directory with ./onedrive or install with make install - but as this is a PR, i would not install

[mickaelleclercq]

After a ./onedrive --logout, and new uri entered, ./onedrive --list-shared-folders was successful !!

$ ./onedrive --list-shared-folders
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...

Listing available OneDrive Business Shared Folders:
---------------------------------------
Shared Folder:   *****
Shared By:       *** *** (****.***@***.***.com)
External Organisation: yes

Bravo !

[abraunegg]

@mickaelleclercq
Thanks for the feedback.

Can you confirm if you can sync the external shared folders or is there an auth / access issue?

[mickaelleclercq]

$ ./onedrive --synchronize --sync-shared-folders --verbose
Using 'user' Config Dir: /home/mickael/.config/onedrive
Using 'system' Config Dir: /etc/onedrive
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /media/mickael/onedrive
Application version: v2.4.8-29-g0e886ac
Account Type: business
Default Drive ID: *****
Default Root ID: *****
Remaining Free Space: 967786905062
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: *******
Updated Remaining Free Space: 967786905062

Processing 201 OneDrive items to ensure consistent local state
Creating local directory: cloud

ERROR: The local file system returned an error with the following message:
  Error Message:    cloud: Permission denied
  Calling Function: applyNewItem()

Processing 206 OneDrive items to ensure consistent local state
Creating local directory: cloud

Error is repeating many times

$ ./onedrive --list-shared-folders
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...

Listing available OneDrive Business Shared Folders:
---------------------------------------
Shared Folder:   CHUQ
Shared By:       ****
External Organisation: yes

$ ./onedrive --display-config
Configuration file successfully loaded
onedrive version                       = v2.4.8-29-g0e886ac
Config path                            = /home/mickael/.config/onedrive
Config file found in config path       = true
Config option 'check_nosync'           = false
Config option 'sync_dir'               = /media/mickael/onedrive
Config option 'skip_dir'               =
Config option 'skip_file'              = ~*|.~*|*.tmp
Config option 'skip_dotfiles'          = false
Config option 'skip_symlinks'          = false
Config option 'monitor_interval'       = 300
Config option 'min_notify_changes'     = 5
Config option 'log_dir'                = /var/log/onedrive/
Config option 'classify_as_big_delete' = 1000
Config option 'upload_only'            = false
Config option 'no_remote_delete'       = false
Config option 'remove_source_files'    = false
Config option 'sync_root_files'        = false
Selective sync 'sync_list' configured  = false
Business Shared Folders configured     = true
business_shared_folders contents:
#comment
CHUQ

[abraunegg]

@mickaelleclercq
Thanks for the feedback. That is the API authentication issue which Microsoft still has to fix. This is what is preventing this feature from being merged / available.

If you look at the debug logs + add --debug-https you will see that the request for auth goes out, and is received, but the access scopes are dropped by Azure.

Because the access scopes were dropped, this is why you get cloud: Permission denied

[mickaelleclercq]

Let's hope that Microsoft work on this soon

[ibrewster]

So, two years later, I assume this has still not been fixed by Microsoft? I'm trying to find some way to upload a file to a shared directory from Linux, and so far this was looking like my best bet...

[abraunegg]

@ibrewster

So, two years later, I assume this has still not been fixed by Microsoft? I'm trying to find some way to upload a file to a shared directory from Linux, and so far this was looking like my best bet...

If the organisation is not yours, then no - if you are part of the organisation you are trying to upload to there should be no issue