The Blender File Open dialog triggers OneDrive Files On-Demand downloads #87621
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#87621
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
System Information
Blender Version
Broken: version: 2.92 - 3.0.0 Alpha
Worked: (To check)
Short description of error
OneDrive has a feature called "Files On-Demand".
This enables the OneDrive folder and its contents to be displayed in the Finder, without actually downloading all files to the local machine.
When you actually access/open a file it gets downloaded to the local machine first and then opened.
The Blender file dialogs trigger this download even when just browsing the OneDrive folder.
So if a OneDrive folder contains files, merely visiting that OneDrive folder using Blender will trigger a download of all those files.
Depending on the size and count of the files that have to be downloaded and the download speed, this also freezes Blender until the downloads have finished.
I noticed this behavior using OneDrive version 21.052.0314.0001.
Other apps and MacOS are perfectly able to navigate/enumerate a OneDrive folder, without downloading its contents.
Update 1: It could be the same problem as described in #87472 (Open file dialog causes application to hang).
Update 2: An possible workaround: on MacOS you can use OneDrive settings to exclude certain apps from triggering OneDrive on-demand downloads.
Exact steps for others to reproduce the error
Note that the images are downloaded at that moment.
Apparently the Thumbnails read by Blender are created and saved in
C:\Users\%user%\.thumbnails\large
.Added subscriber: @Zandman
Added subscriber: @deadpin
Are you using the Thumbnail view in the File Browser? Locally on Windows+OneDrive here the Blender file browser does not download anything unless I toggle on the thumbnail view. At this point blender needs to load the thumbnail picture that's stored in the .blend file, hence the download.
No, not using thumbnail view. Using the default "Files as a vertical list" view. I just noticed OneDrive also displayed a MacOS notification saying that Blender is downloading files from OneDrive. In the OneDrive settings I can excempt Blender from triggering downloads, but I'd rather just have it working properly.
I also noticed your talking about Windows. This seems to be a MacOS issue.
Here's some prior art: https://answers.microsoft.com/en-us/msoffice/forum/msoffice_onedrivefb-mso_mac-mso_o365b/how-to-avoid-unwanted-downloading-of-files-with/39b2de3f-3444-41b5-b988-64c96e5a1838?auth=1
In any case, it seems Blender's file dialog is enumerating files in a way that triggers their download from OneDrive. Fortunately it only does it for files in the currently folder. It doesn't download sub-folders. But when I navigate to a sub-folder, all files within will be downloaded.
It appears this issue is also happening on Windows: #87472 (Open file dialog causes application to hang)
I did some debugging and I've narrowed it down to line 82 in
source/blender/blenlib/intern/storage_apple.mm
:When this line is executed for a file in my OneDrive folder that has not yet been downloaded, it will trigger OneDrive to download the file. When I add a return statement right above line 82 so that the code isn't executed, the file won't be downloaded.
So apparently obtaining info about files whether they are aliases/hidden/readable/writable causes them to be touched in a way that triggers OneDrive. Maybe there's a way to obtain this info without triggering the OneDrive download?
After some more spelunking it appears that the download is triggered specifically for
NSURLIsReadableKey
andNSURLIsWritableKey
. Checking for the other two keys won't trigger a download.Why does the Blender file dialog even need to know whether a file is readable or writable when enumerating them? Is this info actively used at that stage? Or is it gathering this info so add-ons can use it?
Added subscriber: @mano-wii
Please use
Help -> Report a Bug
in Blender so there is complete information about the Blender version and OS.Optionally you could provide the
system-info.txt
generated inHelp -> Save System Info
.Windows and MacOS have a different API for reading files, so it is quite possible that the problem in #87472 is not related.
But it’s a good start to investigate if it’s reproducible on any MacOS.
Changed status from 'Needs Triage' to: 'Confirmed'
I found the steps to reproduce the problem. I can confirm it now.
I suppose it is a high priority as it can negatively affect many users working on OneDrive.
Added subscriber: @ankitm
Because the thumbnail overlays like for non-writable files, aliases depend on that knowledge. The code was added in
1f223b9a1f
andafb1a64ccb
Added subscriber: @Harley
The central problems seems to be that users are asking for thumbnail creation for files that are not fully downloaded. So if this issue is fixed we'll get complaints that users cannot see thumbnails for files in OneDrive. LOL
Sounds like we can just ignore the request to make thumbnails if the the file attributes include whatever OneDrive uses for files in that state. Are you able to attrib some of those files while they are not fully downloaded? I'm guessing it uses FILE_ATTRIBUTE_OFFLINE or FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN or FILE_ATTRIBUTE_SPARSE_FILE or FILE_ATTRIBUTE_TEMPORARY?
I've seen conflicting information about attributes used by OneDrive, including that there might be new ones. FILE_ATTRIBUTE_PINNED, FILE_ATTRIBUTE_UNPINNED, FILE_ATTRIBUTE_RECALL_ON_OPEN, FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS
I've also read that some programs calling regular API functions can get a filtered version of attributes: https://stackoverflow.com/questions/54560454/getting-full-file-attributes-for-files-managed-by-microsoft-onedrive
If treating these files differently by attribute is not feasible then we could always just deal with everything in special folder locations FOLDERID_SkyDrive, FOLDERID_SkyDriveDocuments, FOLDERID_SkyDrivePictures, FOLDERID_SkyDriveMusic, FOLDERID_SkyDriveCameraRoll
I was just experimenting with these OneDrive pinning attributes on my local machine. They've changed the implementation a couple of times during these last years, but I think the current Windows 10 implementation is pretty stable.
However, on Windows I cannot reproduce the unwanted downloads. On my system Blender only triggers file downloads when I switch to thumbnails view. Which kinda makes sense. But on macOS downloads are triggered even when in normal list view. So there is different behavior.
MacOS 10.7 and later also have the concept of files on-demand for iCloud. There are also APIs to query the file state. I have to experiment some more to see how this all works. Also, I don't know if iCloud and OneDrive both use the same implementation of this. I don't think so, as iCloud files seem to be renamed to a hidden counterpart (filename starting with a dot) when cloud-only. I don't see this behavior with OneDrive. Would be nice if both OneDrive and iCloud used the same scheme on the Mac. Maybe they will now that Apple seems to have a formalized way to handle cloud files. Here's some info on how you can query iCloud files: https://developer.apple.com/documentation/foundation/filemanager
To summarize: I guess Blender needs to be aware of cloud-files for both OneDrive and iCloud and handle them appropriately.
In addition to that: currently the Blender file dialog doesn't show the OneDrive or iCloud folders in the left side navigation. There's a case to be made to add this, as both Windows and Mac seem to view them as part of the system.
Yeah, if you want thumbnails of files then they have to be available to load and examine. Doing it "better" in this case would be checking these file attributes to better know their state, and if they are not fully available then not do thumbnailing - despite being in that file view - and show some other type of icon instead perhaps. OneDrive is so integrated, as far as users see, that I'm not sure they would understand why their files were being shown in a different way in this particular place. Some users even change their "My Documents" to use a OneDrive folder. We don't really have control over what they do.
It might be the check for aliasing. With an alias (or shortcut) we want to know the actual target. With Windows we only examine the file to find the target if the file has a (hidden) extension of ".lnk". But Mac aliases don't follow the same type of pattern and it might be possible that the process we use to determine "is this an an alias" has to read the file in a way that requires the file to... be there. Not sure as I don't compile for Mac.
That could be difficult without any kind of standard though. Any way we could deal with OneDrive will not apply to Dropbox or Google Drive, Amazon Drive etc. So there are a lot more of these than the "both." And these will differ between platforms as well. Ideally they would just integrate their systems into the OS in a way that would be transparent to client programs. Or users will learn that these file hosting and synchronization services do have some caveats to be aware of.
Worth considering adding those shortcuts to the "System" lists if they ever work better, but not sure I'd want to encourage their use while just turning on thumbnailing can render them so sucky. When I was adding most of those Windows "special" folders to the "System" list I did consider "OneDrive". But I was loath to add it as it would promote a Microsoft commercial service over other similar services. And then we'd invariably get complaints of "why isn't DropBox shown?"
But the Blender File Manager is actually aware of the location of the OneDrive folder. If you just browse into your user root folder ("Home" icon in the System list) you will see, among the folders for "Desktop", "Music", etc, one for "OneDrive" and that folder will have a special icon if in thumbnail view. And if you add a bookmark while in that folder you will get an item in your "Favorites" list with a special icon too. On Windows anyway.
Some further details.
On Windows, files in this "unavailable" state have FILE_ATTRIBUTE_OFFLINE set. There are other file attributes used for this process:
online-only
FILE_ATTRIBUTE_PINNED : False
FILE_ATTRIBUTE_UNPINNED : True
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS : True
locally available
FILE_ATTRIBUTE_PINNED : False
FILE_ATTRIBUTE_UNPINNED : False
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS : False
always available
FILE_ATTRIBUTE_PINNED : True
FILE_ATTRIBUTE_UNPINNED : False
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS : False
But I'd say that none of these new attributes matter. The only thing that matters for this case is that it is marked as offline, using an attribute that has been in place for years. So also likely to be used by other similar systems.
And we are currently checking and storing this one two. Calling BLI_file_attributes() does include FILE_ATTR_OFFLINE (from BLI_fileops.h eFileAttributes) if FILE_ATTRIBUTE_OFFLINE.
So any time we have a file list entry (FileDirEntry or FileListInternEntry) the attributes member can tell us if this is offline. So hopefully we have one when thumbnails are made, or can perhaps call BLI_file_attributes() if not.
Although there is an Mac version of BLI_file_attributes() in storage_apple.mm it does not populate FILE_ATTRIBUTE_OFFLINE, nor could I quickly see that was an option to do with their API.
So there is almost certainly a one-line fix to this.
editors\space_file\filelist.c has a background job that makes preview thumbnails and saves ids to the resulting thumbnail in preview_icon_id member of FileDirEntry. So we just have to not do thumbnail creation if FileDirEntry.attributes contains FILE_ATTRIBUTE_OFFLINE
Not yet understanding that process yet though.
I'd be inclined towards supporting native cloud drives. I could only find API for iCloud drive on mac, and nothing for OneDrive.
Unfortunately, I think this thread has gotten a bit off track by all the responses about thumbnail generation. This issue isn't related to thumbnail generation. Yes, when Blender generates thumbnails it will trigger the download of those cloud files. However, this I could see as perfectly acceptable behavior, because Blender actually has to read the file for thumbnails to be generated.
The main issue I reported however seems to only happen on macOS, where merely enumerating the files in a OneDrive folder triggers the downloading of those files. And this happens because each file is queried specifically for the
NSURLIsReadableKey
andNSURLIsReadableKey
attributes. I've tested this in a separate application and it can be reproduced. I think this is a bug (or at least an unfortunate side-effect) that has to be fixed. Now this might also be a bug in the OneDrive application. Because querying for those other attributes does not trigger a download. I've contacted the OneDrive team about this, but haven't yet heard back from them. A (temporary) workaround might be to only query for those attributes after a user has explicitly selected the thumbnail view.Now, besides that main issue there's the broader issue of supporting cloud systems like iCloud and OneDrive. And I think those two can be singled out for now, because those two are actually considered system services and by now offer official API support for their respective on-demand files implementation.
Nope, at least not on Mac.
Indeed. But as I've said both Windows and Mac seem to consider their cloud drive solutions to be part of the system and have added explicit API support for such files. So the way I see it Blender running on Mac should natively support iCloud. And Blender running on Windows should natively support OneDrive.
It gets more difficult when Blender has to support non-native cloud drive solutions. In an ideal world all those cloud drive solutions would use the system-provided APIs. That doesn't seem to be the case yet. But that might change in the near future. So OneDrive on macOS has its own Finder integration and on-demand file scheme. But it might use the iCloud way going forward.
As I've said, I consider them system services. So supporting OneDrive on Windows and iCloud on Mac seems perfectly legal to me.
I hope by now you know this isn't about thumbnails and unfortunately there is no one-line fix. Unless the querying for
NSURLIsReadableKey
andNSURLIsReadableKey
gets removed. That worked for me, but I haven't tested the impact of such a change further down the line.Actually you reported multiple issues on both MacOS and Windows. I'm making notes and will probably make changes for the Windows platform. Someone else will have to address the MacOS-related issues.
Created #87867 (macOS: attribute check for "file on demand" OneDrive files triggers download )
Yes, I understand these discussions have surfaced multiple issues on both Windows and Mac. How are you going to track these things? Are you going to create multiple tasks from this task?
Maybe I can fix the Mac issues? I've been looking for a way to contribute to Blender for a while now. For macOS I see the following tasks:
Do you agree with these?
Probably. About to submit a patch that references this one.
Those all sounds like great things to address. There is a task you can reference here: https://developer.blender.org/T87867
This issue was referenced by
ee5ad46a0e
Changed status from 'Confirmed' to: 'Resolved'