PNG format file sequence render corrupt randomly #44521

New Issue

johan tri handoyo · 2015-04-27T12:28:08+02:00

johan tri handoyo commented

2015-04-27 12:28:08 +02:00

System Information
Ubuntu 14.04 64bit, GTX 570

Blender Version
Broken: 2.74
Worked: -

Short description of error
I render the image sequence in .png format using terminal command "./blender -b {scene file} -a" and use multiple terminals and multiple computers to speed up the render time. The folder is on another harddrive which is WD My Book Live 2TB. All computer were connected via NAS. The result is some of the .png files randomly couldn't recognized by nautilus thumbnail and resulting in black preview on Blender VSE after editing. I'm using Blender Render and Approximate Occlusion feature.

I attach the preview thumbnail in nautilus; the "broken" file (file number 0147.png); and the correct png file (file number 0148.png)

**System Information** Ubuntu 14.04 64bit, GTX 570 **Blender Version** Broken: 2.74 Worked: - **Short description of error** I render the image sequence in .png format using terminal command "./blender -b {scene file} -a" and use multiple terminals and multiple computers to speed up the render time. The folder is on another harddrive which is WD My Book Live 2TB. All computer were connected via NAS. The result is some of the .png files randomly couldn't recognized by nautilus thumbnail and resulting in black preview on Blender VSE after editing. I'm using Blender Render and Approximate Occlusion feature. I attach the preview thumbnail in nautilus; the "broken" file (file number 0147.png); and the correct png file (file number 0148.png) ![png_sequence_error.jpg](https://archive.blender.org/developer/F167293/png_sequence_error.jpg) ![0147.png](https://archive.blender.org/developer/F167295/0147.png) ![0148.png](https://archive.blender.org/developer/F167297/0148.png)

johan tri handoyo commented

2015-04-27 12:28:09 +02:00

Changed status to: 'Open'

johan tri handoyo commented

2015-04-27 12:28:09 +02:00

Added subscriber: @JohanTriHandoyo

Bastien Montagne commented

2015-04-27 12:40:40 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2015-04-27 12:40:40 +02:00

I really do not see what we can do with those info… To investigate a bug we need to be able to reproduce it!

Does the issue happens really randomly, or if you re-run the render in the exact same conditions is it always the same pictures which get corrupted?
Does it happen on other system (same render on another PC or so)?

"./blender -b {scene file} -a" and use multiple terminals and multiple computers to speed up the render time

I assume you then manually divide tasks between processes (i.e. process 1 renders frames 1 to 10, process 2 frames 11 to 20, etc.) ?

And quite obviously, a .blend file to help reproducing the issue would also help a lot…

I really do not see what we can do with those info… To investigate a bug we need to be able to reproduce it! * Does the issue happens really randomly, or if you re-run the render in the exact same conditions is it always the same pictures which get corrupted? * Does it happen on other system (same render on another PC or so)? > "./blender -b {scene file} -a" and use multiple terminals and multiple computers to speed up the render time * I assume you then manually divide tasks between processes (i.e. process 1 renders frames 1 to 10, process 2 frames 11 to 20, etc.) ? And quite obviously, a .blend file to help reproducing the issue would also help a lot…

johan tri handoyo commented

2015-04-27 12:47:24 +02:00

this is the file :D https://dl.dropboxusercontent.com/u/13769882/png%20render%20error.zip?dl=1

johan tri handoyo commented

2015-04-27 12:54:25 +02:00

Hi Bastien, to answer your question:

1). Sometimes it's corrupted sometimes it's not. I render it again on frame 147, but everything rendered correctly.
2). I assume that it happened randomly between other system. I use Ubuntu and also windows 7. But it appears when i use Ubuntu the randomly corrupt files occured.
3). Actually i just use Placeholders feature, so i just use one command for all the terminal and Blender will detect unrendered frame automatically.

The .blend file i attach is missing the background models, but it's okay i think.

Hi Bastien, to answer your question: 1). Sometimes it's corrupted sometimes it's not. I render it again on frame 147, but everything rendered correctly. 2). I assume that it happened randomly between other system. I use Ubuntu and also windows 7. But it appears when i use Ubuntu the randomly corrupt files occured. 3). Actually i just use Placeholders feature, so i just use one command for all the terminal and Blender will detect unrendered frame automatically. The .blend file i attach is missing the background models, but it's okay i think.

Bastien Montagne commented

2015-04-27 14:19:32 +02:00

Added subscriber: @Sergey

Bastien Montagne commented

2015-04-27 14:19:32 +02:00

Uuuuhhh… So, you are disabling 'Overwrite' and enabling 'Placeholders', right? Not sure at all this is intended to work this way

Sergey, do you know whether 'overwrite'/'placeholder' handling could be considered “thread safe”? I kind of doubt that. Even worth across a network!

@JohanTriHandoyo suggest you try rendering by 'allocating' specific work to all of your processes, and see whether you still get corruption.

Uuuuhhh… So, you are disabling 'Overwrite' and enabling 'Placeholders', right? Not sure at all this is intended to work this way Sergey, do you know whether 'overwrite'/'placeholder' handling could be considered “thread safe”? I kind of doubt that. Even worth across a network! @JohanTriHandoyo suggest you try rendering by 'allocating' specific work to all of your processes, and see whether you still get corruption.

Sergey Sharybin commented

2015-04-28 00:12:09 +02:00

Added subscriber: @ideasman42

Sergey Sharybin commented

2015-04-28 00:12:09 +02:00

@mont29, i'm not really sure without having a dig into code (which i can do tomorrow). Meanwhile @ideasman42 might know more since this option was heavily used in the bi-farm.

Bastien Montagne self-assigned this 2015-05-06 09:54:56 +02:00

Bastien Montagne commented

2015-05-06 09:54:56 +02:00

Will have a look at that one then.

Bastien Montagne commented

2015-05-06 21:15:51 +02:00

Ok… so, think this is a mere synchronization issue here. Blender does not take any insurance (like a global shared lockfile in render dir or so) that it synchronizes with other processes when handling render files. I think issue does not arise much on a local system (or high speed network one), but probably across a basic NAS (you did not specified used protocol, btw)…

Think what happens is, two different processes check for existence of file at same time, they both find it does not exists, touch it, render, and write result. I’d assume invalid PNG comes from writing at the same time at the end of the render (it’s not the safest thing to do, see also http://stackoverflow.com/questions/7842511/safe-to-have-multiple-processes-writing-to-the-same-file-at-the-same-time-cent).

Note I could not reproduce it here with a single machine, thus assuming issue arises much more easily with 'slow' NAS.

Anyway, not sure we consider that a bug, more like internal limitation? Fixing this would imply adding some filelock system imho…

Ok… so, think this is a mere synchronization issue here. Blender does not take any insurance (like a global shared lockfile in render dir or so) that it synchronizes with other processes when handling render files. I think issue does not arise much on a local system (or high speed network one), but probably across a basic NAS (you did not specified used protocol, btw)… Think what happens is, two different processes check for existence of file at same time, they both find it does not exists, touch it, render, and write result. I’d assume invalid PNG comes from writing at the same time at the end of the render (it’s not the safest thing to do, see also http://stackoverflow.com/questions/7842511/safe-to-have-multiple-processes-writing-to-the-same-file-at-the-same-time-cent). Note I could not reproduce it here with a single machine, thus assuming issue arises much more easily with 'slow' NAS. Anyway, not sure we consider that a bug, more like internal limitation? Fixing this would imply adding some filelock system imho…

johan tri handoyo commented

2015-05-12 13:02:06 +02:00

So i change the LAN cables to a new one, it seems that my current cable is 5+years old :D interesting though, now the frames are all perfectly fine, no more corrupt frames even though i noticed that some of render terminal were rendering the same frame, i was hoping that it will become conflicted and that frame will be corrupted, but it's not.

By the way, i'm using multiple ubuntu terminal to render one shot because the Occlusion Preprocessing is only using 1 thread, and i read in the blenderartist forum that using multiple rendering will speed up the render time.

Thank you for all your help!

So i change the LAN cables to a new one, it seems that my current cable is 5+years old :D interesting though, now the frames are all perfectly fine, no more corrupt frames even though i noticed that some of render terminal were rendering the same frame, i was hoping that it will become conflicted and that frame will be corrupted, but it's not. By the way, i'm using multiple ubuntu terminal to render one shot because the Occlusion Preprocessing is only using 1 thread, and i read in the blenderartist forum that using multiple rendering will speed up the render time. Thank you for all your help!

Bastien Montagne commented

2015-05-17 16:01:13 +02:00

Changed status from 'Open' to: 'Archived'

Bastien Montagne closed this issue

2015-05-17 16:01:13 +02:00

Bastien Montagne commented

2015-05-17 16:01:13 +02:00

Archiving this for now… hard to investigate such issue, especially if we cannot reproduce anyway. I have a bit of a hard time believing a mere too old cable could cause that, but well…

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

PNG format file sequence render corrupt randomly #44521