Official Blender builds for Windows don't recognize NTFS symlinks #37619

Closed
opened 2013-11-25 22:17:41 +01:00 by Ralf Hölzemer · 29 comments

System Information
Operating system and graphics card
Win7 64bit, NVIDIA Quadro 4000, 2 * NVIDIA Geforce 680

Blender Version
Broken: All official Blender Windows builds
Worked: All builds from builder.blender.org

Short description of error
Official Blender builds don't "understand" NTFS symlinks - test builds from builder.blender.org do!
As in UNIX, symlinks in Windows are a very good way to keep paths identical across multiple machines - this is especially useful to integrate Blender into networked Studio workflows.

Symlinks in Windows can per default only be created by the local administrator. Therefor these instructions should be tested with an administrator account or a command prompt window running as administrator. We also need a network share to which the symlink can point to.

In the example below:

  • C:\link is the symlink

  • \server\shareis the targetExact steps for others to reproduce the error
    Based on a (as simple as possible) attached .blend file with minimum amount of steps

  • open a command prompt as administrator

  • mklink /D C:\link \server\share

  • verify in windows explorer that the link is browesable and pointing to the network share

  • open a official release build of blender

  • go to file->open

  • Select the C: drive

The link should now be visible in Blenders file browser, but it isn't

now repeat 4-6, but with a testbuild from builder.blender.org

Link is shown in Blender file browser and behaves correctly as if it was a local directory.

**System Information** Operating system and graphics card Win7 64bit, NVIDIA Quadro 4000, 2 * NVIDIA Geforce 680 **Blender Version** Broken: All official Blender Windows builds Worked: All builds from builder.blender.org **Short description of error** Official Blender builds don't "understand" NTFS symlinks - test builds from builder.blender.org do! As in UNIX, symlinks in Windows are a very good way to keep paths identical across multiple machines - this is especially useful to integrate Blender into networked Studio workflows. Symlinks in Windows can per default only be created by the local administrator. Therefor these instructions should be tested with an administrator account or a command prompt window running as administrator. We also need a network share to which the symlink can point to. **In the example below:** - **C:\link** is the symlink - **\\server\share**is the target**Exact steps for others to reproduce the error** Based on a (as simple as possible) attached .blend file with minimum amount of steps - open a command prompt as administrator - mklink /D C:\link \\server\share - verify in windows explorer that the link is browesable and pointing to the network share - open a official release build of blender - go to file->open - Select the C: drive The link should now be visible in Blenders file browser, but it isn't # now repeat 4-6, but with a testbuild from builder.blender.org Link is shown in Blender file browser and behaves correctly as if it was a local directory.
Author

Changed status to: 'Open'

Changed status to: 'Open'
Author

Added subscribers: @cheleb, @STEP-ANI-MOTION

Added subscribers: @cheleb, @STEP-ANI-MOTION

Added subscriber: @ThomasDinges

Added subscriber: @ThomasDinges

Why is buildbot and official different here? Do you use the "official" builds from the buildbot or the experimental ones?

Either this was fixed after 2.69 or something fishy is going on...

Why is buildbot and official different here? Do you use the "official" builds from the buildbot or the experimental ones? Either this was fixed after 2.69 or something fishy is going on...
Author

Hi dingto,

I really don't know whats different in the buildbot environment. We do use the "official" builds from buildbot - not the experimental mingw or the VS2012 builds.

This is also not restricted to Blender 2.69. As far as I can remember it was always this way. Buildbot versions worked fine, official build from blender.org not.

Hi dingto, I really don't know whats different in the buildbot environment. We do use the "official" builds from buildbot - not the experimental mingw or the VS2012 builds. This is also not restricted to Blender 2.69. As far as I can remember it was always this way. Buildbot versions worked fine, official build from blender.org not.
Author

Quick update,

don't know if it helps. Symlinks are also broken in the VS2012 builds from builder.blender.org, so it looks like the build environment for the "official" builds from builder.b.o is special - they are the only ones that support symlinks.

Quick update, don't know if it helps. Symlinks are also broken in the VS2012 builds from builder.blender.org, so it looks like the build environment for the "official" builds from builder.b.o is special - they are the only ones that support symlinks.

Added subscriber: @Sergey

Added subscriber: @Sergey

Sergey, you know the buildbot well, any idea why this happens? I can't imagine a valid reason...

Sergey, you know the buildbot well, any idea why this happens? I can't imagine a valid reason...

This comment was removed by @ThomasDinges

*This comment was removed by @ThomasDinges*

@ThomasDinges ,not actually. Maybe some specific to particular compiler or runtime library. For my knowledge no custom user-config.py is used or so.

You might contact Jeffrey and ask him, maybe he've got ideas..

@ThomasDinges ,not actually. Maybe some specific to particular compiler or runtime library. For my knowledge no custom user-config.py is used or so. You might contact Jeffrey and ask him, maybe he've got ideas..

Added subscriber: @italic

Added subscriber: @italic

Added subscriber: @brecht

Added subscriber: @brecht

This is all comparing 64 bit builds right? The difference is not in 32/64 bit?

Build logs from buildbot show here with environment variables (you can see same thing locally running 'set' in the console). It could be useful to compare them with what you build locally.
http://builder.blender.org/builders/win64_scons/builds/1260/steps/compile/logs/stdio

This is all comparing 64 bit builds right? The difference is not in 32/64 bit? Build logs from buildbot show here with environment variables (you can see same thing locally running 'set' in the console). It could be useful to compare them with what you build locally. http://builder.blender.org/builders/win64_scons/builds/1260/steps/compile/logs/stdio
Author

Hi @brecht,

yes, as of now, this was all tested with 64bit builds. To get an overview of the situation, I just tested all available Windows builds on b.o and b.b.o. See table below.
Not sure what you mean about building blender locally. This is just about builds available from blender.org/download and builder.blender.org/download.

b.o win32 (2.69) b.o win64 (2.69) b.b.o win32 Official b.b.o win64 Official b.b.o win32 VS2012 b.b.o win64 VS2012
Symlinks work NO NO YES YES NO NO

So it seems the builds on builder.blender.org marked as "Official" are the only ones supporting symlinks under Windows (32 and 64bit).

Hi @brecht, yes, as of now, this was all tested with 64bit builds. To get an overview of the situation, I just tested all available Windows builds on b.o and b.b.o. See table below. Not sure what you mean about building blender locally. This is just about builds available from blender.org/download and builder.blender.org/download. || b.o win32 (2.69) | b.o win64 (2.69)| b.b.o win32 Official | b.b.o win64 Official | b.b.o win32 VS2012 | b.b.o win64 VS2012 | | -- | -- | -- | -- | -- | -- | -- | |Symlinks work | NO | NO | YES | YES | NO | NO | So it seems the builds on builder.blender.org marked as "Official" are the only ones supporting symlinks under Windows (32 and 64bit).

I can't say I have done anything special in setting up my builder. My current Blender setup is entirely run from my B drive, simply to keep it all contained. Buildbot is entirely maintained and run by a separate user, "buildbot." The only caveat was that I didn't allocate enough space on my B drive to have multiple copies of libs; I symlinked a single copy of each lib into their own buildbot directory to save space. Each BB directory has its own source, though.

Otherwise, I use MSVC 2008 Express with the x64 hack listed in the build instructions. I created symlinks with a GUI extension called "Link Shell Extension"
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html
My path does not seem to reveal anything about symlinks.

Regarding recreating the issue:
I have several symlinks set up on my B drive, and I noticed when I went into the official build that this behavior is present. It will not let you browse progressively through the symlinks. However, when the file browser starts in a directory within that symlink, it allows you to browse directories normally (up to a point, as you'll see later). I've uploaded a video illustrating this:
https://www.youtube.com/watch?v=hnW51P8vPyU
The most interesting part is towards the end where Blender is treating a directory within the symlink as root.

I can't say I have done anything special in setting up my builder. My current Blender setup is entirely run from my B drive, simply to keep it all contained. Buildbot is entirely maintained and run by a separate user, "buildbot." The only caveat was that I didn't allocate enough space on my B drive to have multiple copies of libs; I symlinked a single copy of each lib into their own buildbot directory to save space. Each BB directory has its own source, though. Otherwise, I use MSVC 2008 Express with the x64 hack listed in the build instructions. I created symlinks with a GUI extension called "Link Shell Extension" http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html My path does not seem to reveal anything about symlinks. Regarding recreating the issue: I have several symlinks set up on my B drive, and I noticed when I went into the official build that this behavior is present. It will not let you browse progressively through the symlinks. However, when the file browser starts in a directory within that symlink, it allows you to browse directories normally (up to a point, as you'll see later). I've uploaded a video illustrating this: https://www.youtube.com/watch?v=hnW51P8vPyU The most interesting part is towards the end where Blender is treating a directory within the symlink as root.
Author

Thanks @italic for the detailed explanation and the nice video. :)

So it looks like the official b.o release build behaves very similar on your machine to what i see here in our studio. In fact, I just tested the situation where the file browser already starts with a path inside the symlinked directory (thanks for the tip) and that worked here as well until I navigated below the symlink. The symlink would then be displayed as a zero byte file in the file browser and wasn't navigatable anymore.
The weird glitch with blender treating the symlink as root wasn't reproduceable here.

For further illustration, I did some screenshots.

0_Explorer.jpg

  • is the directory containing the symlink (named "network") in windows explorer
  • is the content of that directory
  • shows the properties of the symlink. you can see here that the target (Ziel) is a UNC path (\server\programme)
  • shows the symlink in a command prompt

So on the Windows side, everything is up and running perfectly
1_Blender_Release.jpg

This is the official release from blender.org

  • is the directory containing the symlink (symlink doesn't even show)
  • here I tried to trick the file browser into accepting the path by manually adding the "network" symlink. Blender doesn't know anything about a folder named "network" and offers to create a new one. No thanks :)

... which finally brings us to your builds

2_Blender_Buildbot.jpg

Basically the same tests as above here.

  • is the directory containing the symlink (and it's there \o/)
  • lists the contents of the directory properly

Traversing in- and out of the symlink in this build is no problem at all.

I know this isn't helping very much, but it is essentially all i have for now. Also, we are fine for the time being to use the builds from b.b.o as they usually are fairly stable (thanks for your work on that @italic).

Would be nice to solve this issue for the official builds though as I imagine we are not the only people doing such black magic with symlinks. :)

Thanks @italic for the detailed explanation and the nice video. :) So it looks like the official b.o release build behaves very similar on your machine to what i see here in our studio. In fact, I just tested the situation where the file browser already starts with a path inside the symlinked directory (thanks for the tip) and that worked here as well until I navigated below the symlink. The symlink would then be displayed as a zero byte file in the file browser and wasn't navigatable anymore. The weird glitch with blender treating the symlink as root wasn't reproduceable here. For further illustration, I did some screenshots. ![0_Explorer.jpg](https://archive.blender.org/developer/F33174/0_Explorer.jpg) - is the directory containing the symlink (named "network") in windows explorer - is the content of that directory - shows the properties of the symlink. you can see here that the target (Ziel) is a UNC path (\\server\programme) - shows the symlink in a command prompt So on the Windows side, everything is up and running perfectly ![1_Blender_Release.jpg](https://archive.blender.org/developer/F33176/1_Blender_Release.jpg) This is the official release from blender.org - is the directory containing the symlink (symlink doesn't even show) - here I tried to trick the file browser into accepting the path by manually adding the "network" symlink. Blender doesn't know anything about a folder named "network" and offers to create a new one. No thanks :) ... which finally brings us to your builds ![2_Blender_Buildbot.jpg](https://archive.blender.org/developer/F33178/2_Blender_Buildbot.jpg) Basically the same tests as above here. - is the directory containing the symlink (and it's there \o/) - lists the contents of the directory properly Traversing in- and out of the symlink in this build is no problem at all. I know this isn't helping very much, but it is essentially all i have for now. Also, we are fine for the time being to use the builds from b.b.o as they usually are fairly stable (thanks for your work on that @italic). Would be nice to solve this issue for the official builds though as I imagine we are not the only people doing such black magic with symlinks. :)

@ThomasDinges, @italic any progress in here?

@ThomasDinges, @italic any progress in here?

@Sergey I don't have time to dedicate to personal projects anymore. Whatever is making buildbot respect symlinks is entirely unknown to me; I'm not a coder by any definition. All I can say is that it should keep doing what it's doing. The build environments on the official builder machine is unknown to me. Maybe it has to do with MSVC 2008 express w/ hack versus MSVC 2008 Pro. Another reason might be that symlink explorer extension I use. Something I would try is to build with express w/ hack before installing the extension, then installing it and building again. The extension may change some deeper underlying configuration in Windows to allow easier symlink creation that I'm unaware of. The type of link also makes a difference. The extension allows any number of symlink types, from symlink clones to smart copies to junctions. Someone would have to experiment with each type and see how Blender behaves when it comes across each one (and also create them without the extension through the terminal).

@Sergey I don't have time to dedicate to personal projects anymore. Whatever is making buildbot respect symlinks is entirely unknown to me; I'm not a coder by any definition. All I can say is that it should keep doing what it's doing. The build environments on the official builder machine is unknown to me. Maybe it has to do with MSVC 2008 express w/ hack versus MSVC 2008 Pro. Another reason might be that symlink explorer extension I use. Something I would try is to build with express w/ hack before installing the extension, then installing it and building again. The extension may change some deeper underlying configuration in Windows to allow easier symlink creation that I'm unaware of. The type of link also makes a difference. The extension allows any number of symlink types, from symlink clones to smart copies to junctions. Someone would have to experiment with each type and see how Blender behaves when it comes across each one (and also create them without the extension through the terminal).
Thomas Dinges was assigned by Sergey Sharybin 2014-02-17 10:42:59 +01:00

@ThomasDinges, well you're the windows maintainer, afraid i'm making you responsible to look into the report.

@ThomasDinges, well you're the windows maintainer, afraid i'm making you responsible to look into the report.

Added subscriber: @dragostanasie

Added subscriber: @dragostanasie

Does the issue happe nwit hlatest builds from builder.blender.org?

Does the issue happe nwit hlatest builds from builder.blender.org?

@Sergey
I just checked with the latest 64bit builds from the builder.

  • Doesn't work with the official build.
  • Works with legacy build.
@Sergey I just checked with the latest 64bit builds from the builder. - Doesn't work with the official build. - Works with legacy build.

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'

Spent quite a while on investigation of this issue and in fact it turns out just a missing feature, it only appeared to be looking working because of usage of uninitialzied variable.

The thing is, blender used _wstat64() on windows to distinguish regular files from directories. In old MSVC 2008 it used to fail returning -1 error code and it might leave file information structure uninitialized, which in some cases happened to have "It's a directory flag" and this cases appeared to b working for the reporter. With new MSVC the symlink is considered as a regular file.

To support symbolic links we'll need to read them and implement some sort of stat for the symlink target (to be able to tell whether link points to a dir of file). Now, target in this case is an UNC path which is also not supported by blender, so we'll need t support those first.

Added a note in the TODO list http://wiki.blender.org/index.php/Dev:2.5/Source/Development/Todo/Install-OS#Windows

So thanks for the report, but it's just missing functionality, not a bug.

P.S. You might try mapping UNC path to a driver letter (net use n: \server\folder), this should work i guess.

Spent quite a while on investigation of this issue and in fact it turns out just a missing feature, it only appeared to be looking working because of usage of uninitialzied variable. The thing is, blender used _wstat64() on windows to distinguish regular files from directories. In old MSVC 2008 it used to fail returning -1 error code and it might leave file information structure uninitialized, which in some cases happened to have "It's a directory flag" and this cases appeared to b working for the reporter. With new MSVC the symlink is considered as a regular file. To support symbolic links we'll need to read them and implement some sort of stat for the symlink target (to be able to tell whether link points to a dir of file). Now, target in this case is an UNC path which is also not supported by blender, so we'll need t support those first. Added a note in the TODO list http://wiki.blender.org/index.php/Dev:2.5/Source/Development/Todo/Install-OS#Windows So thanks for the report, but it's just missing functionality, not a bug. P.S. You might try mapping UNC path to a driver letter (net use n: \\server\folder), this should work i guess.

Tried the official win64 builds and it fails, it just does not list the symlink so you cannot browse it.

Unfortunatelly mapping a path to a folder is not a solution if you need relative paths and this lack of functionality prevents Blender from a working network rendering needed by many studios.

The alternative is to implement a modern network rendering for Cycles which is far more complex imho..
Symlinks would be a solid working solution and far easier to implement than Cycles network device rendering.

Tried the official win64 builds and it fails, it just does not list the symlink so you cannot browse it. Unfortunatelly mapping a path to a folder is not a solution if you need relative paths and this lack of functionality prevents Blender from a working network rendering needed by many studios. The alternative is to implement a modern network rendering for Cycles which is far more complex imho.. Symlinks would be a solid working solution and far easier to implement than Cycles network device rendering.

Symlinks are just not supported in blender. And it's not the thing you can easily implement in a proper way - it relaies on such things as making blender aware of UNC paths and so.

I'm not sure why would mapping a share to drive would affect on relative paths.

Symlinks are just not supported in blender. And it's not the thing you can easily implement in a proper way - it relaies on such things as making blender aware of UNC paths and so. I'm not sure why would mapping a share to drive would affect on relative paths.

Hi @Sergey,

first, thank you very much for looking into this issue. We know there are far more pressing issues to solve in blender so it's much appreciated!

Regarding drive letters, we used them prior to win7, when symbolic links to UNC paths weren't possible, but they have their own issues and aren't really an option in our environment anymore. Pretty much anything here depends on symlinks and it would be a huge pita to change that. :)

This functionality relying entirely on a bug in VC2008 is very weird, though it was not as unreliable for us as you described in your previous post. It seems that with wstat64 not returning any error code, blender just assumed an directory entry - which worked absolutely reliable over the last years in our environment. Apparently, there also was/is no need to handle UNC paths in any special way. All of our installed software just works fine that way.

Anyway, as I already said, it is perfectly understandable that this bug/feature is not a high priority thing. We are currently evaluating an in-house build via mingw64, and it looks like it can handle symlinks as it was previously with the unofficial builds from bbo. So I think we might have our temp solution for now.

Hi @Sergey, first, thank you very much for looking into this issue. We know there are far more pressing issues to solve in blender so it's much appreciated! Regarding drive letters, we used them prior to win7, when symbolic links to UNC paths weren't possible, but they have their own issues and aren't really an option in our environment anymore. Pretty much anything here depends on symlinks and it would be a huge pita to change that. :) This functionality relying entirely on a bug in VC2008 is very weird, though it was not as unreliable for us as you described in your previous post. It seems that with wstat64 not returning any error code, blender just assumed an directory entry - which worked absolutely reliable over the last years in our environment. Apparently, there also was/is no need to handle UNC paths in any special way. All of our installed software just works fine that way. Anyway, as I already said, it is perfectly understandable that this bug/feature is not a high priority thing. We are currently evaluating an in-house build via mingw64, and it looks like it can handle symlinks as it was previously with the unofficial builds from bbo. So I think we might have our temp solution for now.

Added subscriber: @Lluc3D

Added subscriber: @Lluc3D

I just found that problem for a project that I'm working one. There is any solution available or hack that we can do to solve the it? I see that the first report was done around 4 years ago, and blender 2.78 still don't recognise linked folders in windows 10.

I just found that problem for a project that I'm working one. There is any solution available or hack that we can do to solve the it? I see that the first report was done around 4 years ago, and blender 2.78 still don't recognise linked folders in windows 10.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
8 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#37619
No description provided.