Junction addon directories have their contents nuked when loadin preferences from previous version #99766
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
5 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#99766
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
System Information
Operating system: Windows 10 x64
Graphics card: NVIDIA GTX 1080
Blender Version
Broken: 3.2.1
Worked: never
Short description of error
When an addon is present in form of a junction, Blender will nuke files and folders in the linked directory in certain situations.
These instructions are specific to the scenario in which I encountered the problem (Moving to new Blender version when debugging Blender addon using Blender VsCode extension ). There probably exist other constellations which cause similar problems related to junctions.
Exact steps for others to reproduce the error
%APPDATA%\Blender Foundation\Blender\3.0\configexists; if necessary, run Blender 3.0.0 once so it is created.%APPDATA%\Blender Foundation\Blender\3.2\configdoes not exist%APPDATA%\Blender Foundation\Blender\3.2\scripts\addonsexistsmklink /J "%APPDATA%\Blender Foundation\Blender\3.2\scripts\addons\myaddon" "path-to-directory-that-will-be-destroyed"mklink /J "%APPDATA%\Blender Foundation\Blender\3.0\scripts\addons\myaddon" "path-to-directory-that-will-be-destroyed"In first-time setup splash screen, click "Load 3.0 Settings".
After this, every file in the folder structure of
path-to-directory-that-will-be-destroyedwill be truncated to 0 bytes in size.Perhaps it would even be sufficient for the 3.0
addonsdirectory to contain a normal directory by the same name as the junction in the 3.2addonsdirectory instead of having to also be a junction itself.A mitigated variant of this problem is encountered when using the addon "Remove" button of an addon in the addon preferences window, where only an OSError is thrown.
Added subscriber: @zyl
Added subscriber: @OmarEmaraDev
Changed status from 'Needs Triage' to: 'Needs User Info'
Can you show reproduction steps without VS Code or the linked VS Code extension? Maybe demonstrate it using the remove addon button as you described?
I am not on Windows, so can't reproduce, But I suspect the following lines in the operation can explain the behavior:
Which behaves differently on Windows.
Seems that the behavior of the "Remove addon"-button has been changed at some point. Since at least 3.0.0 this causes "OSError: Cannot call rmtree on a symbolic link" and only disables the addon, which already is much less critical than it was in former versions. (Probably should be improved to just remove the symlink instead)
So that leaves only the splash screen "Load 3.0 Settings"-button to copy settings from previous installation as main culprit.
I deleted
%APPDATA%\Blender Foundation\Blender\3.2directory and tried again. Things are fine if I don't click the button. So I tried deleting onlyC:\Users\Zyl\AppData\Roaming\Blender Foundation\Blender\3.2\configand now the dialog will show again when starting Blender, again showing the button which causes bad things to happen when clicked.Clicking the button then causes all of the bad stuff to happen and prints this to console:
It is worth noting that another addon which also existed in symlinked form in the settings which were copied (but not in the copy target) was copied successfully, in "normal", non-symlinked manner.
Even more interestingly, in a third attempt, I removed the
.gitdirectory from the project before clicking the "Load 3.0 Settings" button. In that case, the button throws no error, but still nukes the directory contents.So, my complete reproduction steps taking the VsCode extension out of the equation would be:
%APPDATA%\Blender Foundation\Blender\3.0\configexists; if necessary, install and run Blender 3.0.0 once so it is created.%APPDATA%\Blender Foundation\Blender\3.2\scripts\addons.I suspect Blender tries to clean up everything before copy, which is in itself not wrong, but contains what causes the problem. Presumably "Remove addon"-button and "Load 3.0 Settings"-button use two different code paths for addon removal functionality.
I believe these behaviors need to be fixed:
rmtree().I think it would be better if the VsCode extension was changed to not use soft links, but it's not being developed too actively (I reported the problem there in 2019), so eliminating the problem on the Blender side of things would still be a big win for unsuspecting addon developers.
I also tested
os.path.islink()on a soft link:It seems they just didn't bother implementing it for Windows, and always return
Falsethere. The docs confirm this: https://docs.python.org/3/library/os.path.html#os.path.islinkhttps://stackoverflow.com/questions/15258506/os-path-islink-on-windows-with-python has answers proposing alternatives.
Changed status from 'Needs User Info' to: 'Needs Triage'
I tried to do synthetic tests on Linux but failed to reproduce the issue, so I will leave this to someone on Windows.
Added subscriber: @iss
Changed status from 'Needs Triage' to: 'Needs User Info'
@zyl I was unable to reproduce this issue.
Is using admin-privileged terminal in step 4 necessary to reproduce? I assumed it's not as it would be quite strange. But perhaps it is important to run Blender with admin rights?
When using my normal user account to run the command I get "You do not have sufficient privilege to perform this operation.". I find this strange since I can create and modify files and directories in both the source and target location without any such issue. It seems the Windows privilege model grows ever more confusing.
Have you tried to debug the return value of
os.path.islink()? Does it actually returnTrueon your machine?Have you installed the Windows 10 Version 21H2 update? (I have not yet done this, though I would expect for this to be indifferent to the circumstance)
I have not run Blender with admin rights in encountering this issue.
Changed status from 'Needs User Info' to: 'Needs Triage'
Did not check this. Do you suggest that files are deleted by Python itself or Blender script? If it's done by script, can you identify where file is deleted exactly?
Will have to re-check this next week, So meanwhile, will change status back.
Added subscriber: @3di
Just happened to me. Lost two weeks worth of commits as it overwrote an addon which was in a shared folder in the 3.3/scripts/addons/ folder. The shared folder pointed to the location of the master script, and was shared due to it being set up by jacques luckes vs code plugin.
found this, seems as though it might work. Not sure how to identify a root object from the massive list of files in the objects folder and it's sub directories.
gist.github.com/jimrubenstein/9827718c508b2b7f5bad578bae421bfc
sorted. init git repo in empty folder, copy in objects folder from .git folder in the Blender nuked repo. git fsck, then merge the sha1 of the dangling commit.
So I have checked this on Windows 10
And Blender's python says:
Same happens when using relative path, so not sure what the problem is on your side. I assume, you are using python, that is bundled with Blender, is that correct?
Why would the objects folder not be affected by this? All files in the repository are set to 0kb, but the contents of the objects folder is left intact (thankfully!). I believe this is the only folder which git commands never touch, so perhaps it's related to the git related vs code plugins?
I also had vs code in admin mode with the following addons:
I can't remember which version of python I had set as the interpreter, possibly 3.8 as I'd been working in anaconda on some other stuff.
just wondering though, why does it ever need to remove anything if it's copying settings from one folder to another?
shutil.rmtree(path)
os.remove(path)```
I figured it out, and my reproduction steps have missed this: there exists a legacy Windows feature known as "Junctions". The correct command to create one should use the
/Jinstead of the/Dswitch, like so:When using the
dircommand it will then sayJUNCTIONinstead ofSYMLINKD. It seems thatos.path.islink()is oblivious to this. The autocompletion in the Blender Python prompt seems to be unaware of objects of this type as well.I have verified the symlinks created by the VsCode extension originally mentioned are in fact junctions, and not symbolic links. First Google hit: https://ourtechroom.com/tech/junction-vs-symbolic-links/
It seems heightened permissions being required to create symlinks is one of the reasons why the VsCode extension uses them in the first place, which is a bummer because symlinks were added to Windows in part specifically to improve Unix interoperability. I guess having Blender learn how to do deal with junctions would be required to be able to address the problem properly.
I have re-checked and still wasn't able to reproduce this.
Trying to remove junction with
shutil.rmtree(path), as it would be done, resulted in exception:os.remove(path)does remove the junction, but does not touch the content.Perhaps different code is causing this, but it's still strange, that I can't reproduce this issue.
Were you checking the "Remove Addon"-button? The problem has already been mitigated for it. The main problem remains with the first-launch splash screen. See the first two sentences of my first comment (tread with care as I may have used the word symlink when I was actually dealing with a junction).
Yep, it's the splashscreen that caused the issue for me too. No idea why it needs to remove anything if it's just copying from the previous blender version's folder to the new blender version's folder.
I have updated the task description to clarify that the "Remove addon" button does not exhibit the exact same behavior.
Here are new reproduction steps which correctly use the
/Jinstead of the/Dswitch:%APPDATA%\Blender Foundation\Blender\3.0\configexists; if necessary, install and run Blender 3.0.0 once so it is created.%APPDATA%\Blender Foundation\Blender\3.2\scripts\addons.What button is this? is it somewhere in preferences?
I have been following these steps - have junction, splash screen in 3.2.1 and click on Load 3.0 Settings.
Though I have skipped first step as I already have 3.0 folder initialized. Is this correct to do, or do I have to re-create it with this step as outlined?
It's the one which says "Remove" here:
It is sufficient if it exists. I figured out what is missing. With the VsCode extension the scenario was that the junction to the folder would exist in BOTH the
addonsfolder of Blender 3.0 and Blender 3.2 prior to the 3.2 instance being launched.I have updated the original task description with new steps to reproduce.
Symlinked addon directories have their contents nuked in certain situationsto Junction addon directories have their contents nuked in certain situationsJunction addon directories have their contents nuked in certain situationsto Junction addon directories have their contents nuked when loadin preferences from previous versionChanged status from 'Needs Triage' to: 'Confirmed'
Raising priority to high, since there was another report where this issue caused loss of data.
There is not much logic on our end,
shutil.copytreeis called, which doesn't handle junctions overly well, extracted a minimal repro, and reported upstream cpython #104046Given this ticket is 10 months old, the issue is caused by a third party, and appears to be a bug in
shutil.copytreelowering the prio on this, as there's not much to do on the blender side of things.