Saving blend-files straight to server nets 0.5% speed of a windows file copy to the same location #69206

New Issue

Wouter Vandenneucker · 2019-08-27T12:08:14+02:00

2019-08-27 12:08:14 +02:00

System Information
Operating system: Windows 10 (client), Ubuntu 18.04 (client), Windows Server 2019 (server)
Graphics card: GTX 1080 TI, Intel integrated Graphics 4000

Blender Version
Broken: version: 2.80 (sub 75), branch: master, commit date: 2019-07-29 14:47, hash: f6cb5f5449

Short description of error
This is a crosspost of a question I asked at Blender Stackexchange . More specific information and context can be found there.

In short, copying a 2.2GB blend file to a windows share through explorer takes 23s (765Mbps) on our Gigabit network.
Saving the same file to the same share location directly from blender takes +5000s (<3.5Mbps).
The process of saving in itself when writing to the internal SSD takes 4 seconds.

I tried this to multiple shares (albeit on the same server) and from different (wired) clients.

Technically it works, it does save the file, it's just not usable as it is and prevents us from having a better workflow.

Exact steps for others to reproduce the error

Take any blendfile with a considerable size
Try to save to a Windows share
Wait for the process to complete

**System Information** Operating system: Windows 10 (client), Ubuntu 18.04 (client), Windows Server 2019 (server) Graphics card: GTX 1080 TI, Intel integrated Graphics 4000 **Blender Version** Broken: version: 2.80 (sub 75), branch: master, commit date: 2019-07-29 14:47, hash: `f6cb5f5449` **Short description of error** This is a crosspost of a question I asked at [Blender Stackexchange ](https://blender.stackexchange.com/questions/149926/saving-blend-files-straight-to-server-nets-0-5-speed-of-a-windows-file-copy-ho). More specific information and context can be found there. In short, copying a 2.2GB blend file to a windows share through explorer takes 23s (765Mbps) on our Gigabit network. Saving the same file to the same share location directly from blender takes +5000s (<3.5Mbps). The process of saving in itself when writing to the internal SSD takes 4 seconds. I tried this to multiple shares (albeit on the same server) and from different (wired) clients. Technically it works, it does save the file, it's just not usable as it is and prevents us from having a better workflow. **Exact steps for others to reproduce the error** - Take any blendfile with a considerable size - Try to save to a Windows share - Wait for the process to complete

Wouter Vandenneucker commented

2019-08-27 12:08:14 +02:00

Added subscriber: @WouterVandenneucker

Duarte Farrajota Ramos commented

2019-08-27 14:02:42 +02:00

Added subscriber: @DuarteRamos

Duarte Farrajota Ramos commented

2019-08-27 14:02:42 +02:00

Some comments on this: I don't think it is 2.8 related, 2.79 and previous were equally slow.

I noticed in the past that saving directly to a USB connected thumb drive can also be very slow. My guess is this has something to do with the way Blender saves files, not necessarily to network or 2.8 specifically.

Some comments on this: I don't think it is 2.8 related, 2.79 and previous were equally slow. I noticed in the past that saving directly to a USB connected thumb drive can also be very slow. My guess is this has something to do with the way Blender saves files, not necessarily to network or 2.8 specifically.

Wouter Vandenneucker commented

2019-08-27 14:05:16 +02:00

Indeed, I just tested in 2.8 to confirm the issue is still there.

Bastien Montagne commented

2019-08-29 11:27:17 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2019-08-29 11:27:17 +02:00

Changed status from 'Open' to: 'Archived'

Bastien Montagne closed this issue

2019-08-29 11:27:17 +02:00

Bastien Montagne self-assigned this 2019-08-29 11:27:17 +02:00

Bastien Montagne commented

2019-08-29 11:27:17 +02:00

that is almost certainly because copying the file from windows file browser will result in a single write operation of 2.2GB. Blender writes small chunks of memory at a time (among other reasons, to avoid having to store GB's of binary file in RAM...), so on such a big file it most likely will result in thousands of write commands. there are most certainly ways to setup your remote shared drive in a way that those get packed together, but that is really not a Blender bug or issue. This is sysadmin work to find a proper solution here, if you want direct random read/write on a remote storage.

Wouter Vandenneucker commented

2019-08-29 13:34:45 +02:00

Changed status from 'Archived' to: 'Open'

Wouter Vandenneucker reopened this issue

2019-08-29 13:34:45 +02:00

Wouter Vandenneucker commented

2019-08-29 13:34:45 +02:00

I wouldn't say that this is for sysadmin to work out; the 2.2GB file was to even out network delays. Writing any blender project bigger than that standard cube to the network is problematic at this point.

If 2.8 is supposed to be used in production by real studios, direct writing to external filesystems is a real use case in a normal workflow.
Take any other professional software suite, be it an Adobe program, an Autodesk program, Solidworks,... they all handle writing to the network as it's supposed to. Even for +GB filesizes.
It's not a job of a sysadmin to find a way around the programs poor handling of write ops; at least not if the aim is to be a professional software suite.

Even with linking and a proper setup, writing a file of 25MB takes an excessive amount of time to be practical for any professional work.
I can solve this easily by having an rsync tool in the background that uploads everything to right folder, that is besides the point.

Blender should either handle writes to external sources properly or prohibit writing to it as a whole. Having the UI locked for +10 minutes is a definite no-no whichever way you look at it!

I wouldn't say that this is for sysadmin to work out; the 2.2GB file was to even out network delays. Writing any blender project bigger than that standard cube to the network is problematic at this point. If 2.8 is supposed to be used in production by real studios, direct writing to external filesystems is a real use case in a normal workflow. Take any other professional software suite, be it an Adobe program, an Autodesk program, Solidworks,... they all handle writing to the network as it's supposed to. Even for +GB filesizes. It's not a job of a sysadmin to find a way around the programs poor handling of write ops; at least not if the aim is to be a professional software suite. Even with linking and a proper setup, writing a file of 25MB takes an excessive amount of time to be practical for any professional work. I can solve this easily by having an rsync tool in the background that uploads everything to right folder, that is besides the point. Blender should either handle writes to external sources properly or prohibit writing to it as a whole. Having the UI locked for +10 minutes is a definite no-no whichever way you look at it!

Bastien Montagne commented

2019-08-29 14:25:17 +02:00

Changed status from 'Open' to: 'Archived'

Bastien Montagne closed this issue

2019-08-29 14:25:17 +02:00

Bastien Montagne commented

2019-08-29 14:25:17 +02:00

We do not have control over where you write a file> Improvements are always possible, but that is a feature request then, not a bug, and it has nothing to do on this tracker... Also please do not reopen tasks, triaging is to be done by developers, not users.

Robert S commented

2021-04-26 10:10:25 +02:00

Added subscriber: @RobertS

Robert S commented

2021-04-26 10:10:25 +02:00

In #69206#765020, @mont29 wrote:
that is almost certainly because copying the file from windows file browser will result in a single write operation of 2.2GB. Blender writes small chunks of memory at a time (among other reasons, to avoid having to store GB's of binary file in RAM...), so on such a big file it most likely will result in thousands of write commands. there are most certainly ways to setup your remote shared drive in a way that those get packed together, but that is really not a Blender bug or issue. This is sysadmin work to find a proper solution here, if you want direct random read/write on a remote storage.

I see this as well, because we have trouble with speeds over VPN at work. Interestingly, though when exporting a .glb file from Blender, I get the "full" 80mpbs write speed. Only when doing save in the native .blend format do I get 4mbps.

Interestingly, I've measured similar slow save times in other 3D apps, but when I imported the glb into a large Powerpoint presentation and saved that, it also saved at full speed...

> In #69206#765020, @mont29 wrote: > that is almost certainly because copying the file from windows file browser will result in a single write operation of 2.2GB. Blender writes small chunks of memory at a time (among other reasons, to avoid having to store GB's of binary file in RAM...), so on such a big file it most likely will result in thousands of write commands. there are most certainly ways to setup your remote shared drive in a way that those get packed together, but that is really not a Blender bug or issue. This is sysadmin work to find a proper solution here, if you want direct random read/write on a remote storage. I see this as well, because we have trouble with speeds over VPN at work. Interestingly, though when exporting a .glb file from Blender, I get the "full" 80mpbs write speed. Only when doing save in the native .blend format do I get 4mbps. Interestingly, I've measured similar slow save times in other 3D apps, but when I imported the glb into a large Powerpoint presentation and saved that, it also saved at full speed...

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Saving blend-files straight to server nets 0.5% speed of a windows file copy to the same location #69206