Page MenuHome

Official Blender builds for Windows don't recognize NTFS symlinks
Closed, ArchivedPublic

Description

System Information
Operating system and graphics card
Win7 64bit, NVIDIA Quadro 4000, 2 * NVIDIA Geforce 680

Blender Version
Broken: All official Blender Windows builds
Worked: All builds from builder.blender.org

Short description of error
Official Blender builds don't "understand" NTFS symlinks - test builds from builder.blender.org do!
As in UNIX, symlinks in Windows are a very good way to keep paths identical across multiple machines - this is especially useful to integrate Blender into networked Studio workflows.

Symlinks in Windows can per default only be created by the local administrator. Therefor these instructions should be tested with an administrator account or a command prompt window running as administrator. We also need a network share to which the symlink can point to.

In the example below:

  • C:\link is the symlink
  • \\server\share is the target

Exact steps for others to reproduce the error
Based on a (as simple as possible) attached .blend file with minimum amount of steps

  1. open a command prompt as administrator
  2. mklink /D C:\link \\server\share
  3. verify in windows explorer that the link is browesable and pointing to the network share
  4. open a official release build of blender
  5. go to file->open
  6. Select the C: drive

The link should now be visible in Blenders file browser, but it isn't

  1. now repeat 4-6, but with a testbuild from builder.blender.org

Link is shown in Blender file browser and behaves correctly as if it was a local directory.

Details

Type
Bug

Event Timeline

Ralf Hölzemer (cheleb) raised the priority of this task from to Needs Triage by Developer.
Ralf Hölzemer (cheleb) updated the task description. (Show Details)
Ralf Hölzemer (cheleb) set Type to Bug.

Why is buildbot and official different here? Do you use the "official" builds from the buildbot or the experimental ones?

Either this was fixed after 2.69 or something fishy is going on...

Hi dingto,

I really don't know whats different in the buildbot environment. We do use the "official" builds from buildbot - not the experimental mingw or the VS2012 builds.

This is also not restricted to Blender 2.69. As far as I can remember it was always this way. Buildbot versions worked fine, official build from blender.org not.

Quick update,

don't know if it helps. Symlinks are also broken in the VS2012 builds from builder.blender.org, so it looks like the build environment for the "official" builds from builder.b.o is special - they are the only ones that support symlinks.

Sergey, you know the buildbot well, any idea why this happens? I can't imagine a valid reason...

This comment has been deleted.

@Thomas Dinges (dingto) ,not actually. Maybe some specific to particular compiler or runtime library. For my knowledge no custom user-config.py is used or so.

You might contact Jeffrey and ask him, maybe he've got ideas..

This is all comparing 64 bit builds right? The difference is not in 32/64 bit?

Build logs from buildbot show here with environment variables (you can see same thing locally running 'set' in the console). It could be useful to compare them with what you build locally.
http://builder.blender.org/builders/win64_scons/builds/1260/steps/compile/logs/stdio

Brecht Van Lommel (brecht) lowered the priority of this task from Needs Triage by Developer to Normal.Nov 26 2013, 3:37 PM

Hi @Brecht Van Lommel (brecht),

yes, as of now, this was all tested with 64bit builds. To get an overview of the situation, I just tested all available Windows builds on b.o and b.b.o. See table below.
Not sure what you mean about building blender locally. This is just about builds available from blender.org/download and builder.blender.org/download.

b.o win32 (2.69)b.o win64 (2.69)b.b.o win32 Officialb.b.o win64 Officialb.b.o win32 VS2012b.b.o win64 VS2012
Symlinks workNONOYESYESNONO

So it seems the builds on builder.blender.org marked as "Official" are the only ones supporting symlinks under Windows (32 and 64bit).

I can't say I have done anything special in setting up my builder. My current Blender setup is entirely run from my B drive, simply to keep it all contained. Buildbot is entirely maintained and run by a separate user, "buildbot." The only caveat was that I didn't allocate enough space on my B drive to have multiple copies of libs; I symlinked a single copy of each lib into their own buildbot directory to save space. Each BB directory has its own source, though.

Otherwise, I use MSVC 2008 Express with the x64 hack listed in the build instructions. I created symlinks with a GUI extension called "Link Shell Extension"
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html
My path does not seem to reveal anything about symlinks.

Regarding recreating the issue:
I have several symlinks set up on my B drive, and I noticed when I went into the official build that this behavior is present. It will not let you browse progressively through the symlinks. However, when the file browser starts in a directory within that symlink, it allows you to browse directories normally (up to a point, as you'll see later). I've uploaded a video illustrating this:
https://www.youtube.com/watch?v=hnW51P8vPyU
The most interesting part is towards the end where Blender is treating a directory within the symlink as root.

Thanks @Jeffrey Hoover (italic_) for the detailed explanation and the nice video. :)

So it looks like the official b.o release build behaves very similar on your machine to what i see here in our studio. In fact, I just tested the situation where the file browser already starts with a path inside the symlinked directory (thanks for the tip) and that worked here as well until I navigated below the symlink. The symlink would then be displayed as a zero byte file in the file browser and wasn't navigatable anymore.
The weird glitch with blender treating the symlink as root wasn't reproduceable here.

For further illustration, I did some screenshots.

  1. is the directory containing the symlink (named "network") in windows explorer
  2. is the content of that directory
  3. shows the properties of the symlink. you can see here that the target (Ziel) is a UNC path (\\server\programme)
  4. shows the symlink in a command prompt

So on the Windows side, everything is up and running perfectly

This is the official release from blender.org

  1. is the directory containing the symlink (symlink doesn't even show)
  2. here I tried to trick the file browser into accepting the path by manually adding the "network" symlink. Blender doesn't know anything about a folder named "network" and offers to create a new one. No thanks :)

... which finally brings us to your builds

Basically the same tests as above here.

  1. is the directory containing the symlink (and it's there \o/)
  2. lists the contents of the directory properly

Traversing in- and out of the symlink in this build is no problem at all.

I know this isn't helping very much, but it is essentially all i have for now. Also, we are fine for the time being to use the builds from b.b.o as they usually are fairly stable (thanks for your work on that @Jeffrey Hoover (italic_)).

Would be nice to solve this issue for the official builds though as I imagine we are not the only people doing such black magic with symlinks. :)

@Sergey Sharybin (sergey) I don't have time to dedicate to personal projects anymore. Whatever is making buildbot respect symlinks is entirely unknown to me; I'm not a coder by any definition. All I can say is that it should keep doing what it's doing. The build environments on the official builder machine is unknown to me. Maybe it has to do with MSVC 2008 express w/ hack versus MSVC 2008 Pro. Another reason might be that symlink explorer extension I use. Something I would try is to build with express w/ hack before installing the extension, then installing it and building again. The extension may change some deeper underlying configuration in Windows to allow easier symlink creation that I'm unaware of. The type of link also makes a difference. The extension allows any number of symlink types, from symlink clones to smart copies to junctions. Someone would have to experiment with each type and see how Blender behaves when it comes across each one (and also create them without the extension through the terminal).

@Thomas Dinges (dingto), well you're the windows maintainer, afraid i'm making you responsible to look into the report.

Does the issue happe nwit hlatest builds from builder.blender.org?

@Sergey Sharybin (sergey)
I just checked with the latest 64bit builds from the builder.

-Doesn't work with the official build.
-Works with legacy build.

Sergey Sharybin (sergey) closed this task as Archived.Jun 3 2014, 5:00 PM

Spent quite a while on investigation of this issue and in fact it turns out just a missing feature, it only appeared to be looking working because of usage of uninitialzied variable.

The thing is, blender used _wstat64() on windows to distinguish regular files from directories. In old MSVC 2008 it used to fail returning -1 error code and it might leave file information structure uninitialized, which in some cases happened to have "It's a directory flag" and this cases appeared to b working for the reporter. With new MSVC the symlink is considered as a regular file.

To support symbolic links we'll need to read them and implement some sort of stat for the symlink target (to be able to tell whether link points to a dir of file). Now, target in this case is an UNC path which is also not supported by blender, so we'll need t support those first.

Added a note in the TODO list http://wiki.blender.org/index.php/Dev:2.5/Source/Development/Todo/Install-OS#Windows

So thanks for the report, but it's just missing functionality, not a bug.

P.S. You might try mapping UNC path to a driver letter (net use n: \\server\folder), this should work i guess.

Tried the official win64 builds and it fails, it just does not list the symlink so you cannot browse it.

Unfortunatelly mapping a path to a folder is not a solution if you need relative paths and this lack of functionality prevents Blender from a working network rendering needed by many studios.

The alternative is to implement a modern network rendering for Cycles which is far more complex imho..
Symlinks would be a solid working solution and far easier to implement than Cycles network device rendering.

Symlinks are just not supported in blender. And it's not the thing you can easily implement in a proper way - it relaies on such things as making blender aware of UNC paths and so.

I'm not sure why would mapping a share to drive would affect on relative paths.

Hi @Sergey Sharybin (sergey),

first, thank you very much for looking into this issue. We know there are far more pressing issues to solve in blender so it's much appreciated!

Regarding drive letters, we used them prior to win7, when symbolic links to UNC paths weren't possible, but they have their own issues and aren't really an option in our environment anymore. Pretty much anything here depends on symlinks and it would be a huge pita to change that. :)

This functionality relying entirely on a bug in VC2008 is very weird, though it was not as unreliable for us as you described in your previous post. It seems that with wstat64 not returning any error code, blender just assumed an directory entry - which worked absolutely reliable over the last years in our environment. Apparently, there also was/is no need to handle UNC paths in any special way. All of our installed software just works fine that way.

Anyway, as I already said, it is perfectly understandable that this bug/feature is not a high priority thing. We are currently evaluating an in-house build via mingw64, and it looks like it can handle symlinks as it was previously with the unofficial builds from bbo. So I think we might have our temp solution for now.

I just found that problem for a project that I'm working one. There is any solution available or hack that we can do to solve the it? I see that the first report was done around 4 years ago, and blender 2.78 still don't recognise linked folders in windows 10.