Heavy Exr file will not generate previews in asset broswer/file broswer #97810

New Issue

yonghao lv · 2022-05-03T13:59:22+02:00

yonghao lv commented

2022-05-03 13:59:22 +02:00

System Information
Operating system: Windows-10-10.0.22000-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 2080 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.15

Blender Version
Broken: version: 3.3.0 Alpha, branch: master, commit date: 2022-05-02 23:49, hash: e5a738af6d

Short description of error
when using a big exr file (20mb+) , it will not generate thumbs in file broswer
also, the small files behind it will not generate, too

Exact steps for others to reproduce the error
use a folder that contain 30file hdr(1 of them is bigger than 20m, stay in the middle)
open the file brower or assgin the big one to asset preview

However, I use this scripts running in the background to resize the big exr, it will not generate the big files' resize result, but the small ones before a big one

import bpy
import os
from subprocess import run

filepath = '' # source
out_jpg = '' # out thumb
scripts_path = ''

cmd = [bpy.app.binary_path]
cmd.append("--background")
cmd.append("--factory-startup")
cmd.append("--python")
cmd.append(scripts_path)
cmd.append('--')
cmd.append(filepath)
cmd.append(self.resolution)
cmd.append(out_jpg)
run(cmd)

def main(argv):
    FILEPATH, SIZE_X, OUTPATH = argv
    SIZE_X = int(SIZE_X)

    context = bpy.context
    scene = context.scene

    scene.use_nodes = True
    node_tree = scene.node_tree

    # Remove default nodes, except composite
    n_comp = None
    for n in node_tree.nodes:
        if not n.type == 'COMPOSITE':
            node_tree.nodes.remove(n)
        else:
            n_comp = n

    img = bpy.data.images.load(FILEPATH)
    n_img = node_tree.nodes.new("CompositorNodeImage")
    n_img.image = img

    n_blur = node_tree.nodes.new("CompositorNodeBlur")
    n_blur.filter_type = 'FLAT'
    n_blur.size_x = floor(img.size[0] / SIZE_X / 2)
    n_blur.size_y = n_blur.size_x

    n_scale = node_tree.nodes.new("CompositorNodeScale")
    n_scale.space = "RENDER_SIZE"
    n_scale.frame_method = "CROP"

    # Links
    links = node_tree.links
    links.new(n_img.outputs[0], n_blur.inputs[0])
    links.new(n_blur.outputs[0], n_scale.inputs[0])
    links.new(n_scale.outputs[0], n_comp.inputs[0])

    # Render
    r = scene.render
    r.image_settings.file_format = 'JPEG'
    r.image_settings.quality = 95
    r.resolution_x = SIZE_X
    SIZE_Y = floor(SIZE_X / (img.size[0] / img.size[1]))
    r.resolution_y = SIZE_Y
    r.resolution_percentage = 100
    r.filepath = OUTPATH

    bpy.ops.render.render(write_still=True)


if __name__ == "__main__":
    if "--" not in sys.argv:
        argv = []  # as if no args are passed
    else:
        argv = sys.argv[sys.argv.index("--") + 1:]  # get all args after "--"
    main(argv)

**System Information** Operating system: Windows-10-10.0.22000-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 2080 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.15 **Blender Version** Broken: version: 3.3.0 Alpha, branch: master, commit date: 2022-05-02 23:49, hash: `e5a738af6d` **Short description of error** when using a big exr file (20mb+) , it will not generate thumbs in file broswer also, the small files behind it will not generate, too ![image.png](https://archive.blender.org/developer/F13049053/image.png) ![image.png](https://archive.blender.org/developer/F13049068/image.png) **Exact steps for others to reproduce the error** use a folder that contain 30file hdr(1 of them is bigger than 20m, stay in the middle) open the file brower or assgin the big one to asset preview However, I use this scripts running in the background to resize the big exr, it will not generate the big files' resize result, but the small ones before a big one ``` import bpy import os from subprocess import run filepath = '' # source out_jpg = '' # out thumb scripts_path = '' cmd = [bpy.app.binary_path] cmd.append("--background") cmd.append("--factory-startup") cmd.append("--python") cmd.append(scripts_path) cmd.append('--') cmd.append(filepath) cmd.append(self.resolution) cmd.append(out_jpg) run(cmd) ``` ``` def main(argv): FILEPATH, SIZE_X, OUTPATH = argv SIZE_X = int(SIZE_X) context = bpy.context scene = context.scene scene.use_nodes = True node_tree = scene.node_tree # Remove default nodes, except composite n_comp = None for n in node_tree.nodes: if not n.type == 'COMPOSITE': node_tree.nodes.remove(n) else: n_comp = n img = bpy.data.images.load(FILEPATH) n_img = node_tree.nodes.new("CompositorNodeImage") n_img.image = img n_blur = node_tree.nodes.new("CompositorNodeBlur") n_blur.filter_type = 'FLAT' n_blur.size_x = floor(img.size[0] / SIZE_X / 2) n_blur.size_y = n_blur.size_x n_scale = node_tree.nodes.new("CompositorNodeScale") n_scale.space = "RENDER_SIZE" n_scale.frame_method = "CROP" # Links links = node_tree.links links.new(n_img.outputs[0], n_blur.inputs[0]) links.new(n_blur.outputs[0], n_scale.inputs[0]) links.new(n_scale.outputs[0], n_comp.inputs[0]) # Render r = scene.render r.image_settings.file_format = 'JPEG' r.image_settings.quality = 95 r.resolution_x = SIZE_X SIZE_Y = floor(SIZE_X / (img.size[0] / img.size[1])) r.resolution_y = SIZE_Y r.resolution_percentage = 100 r.filepath = OUTPATH bpy.ops.render.render(write_still=True) if __name__ == "__main__": if "--" not in sys.argv: argv = [] # as if no args are passed else: argv = sys.argv[sys.argv.index("--") + 1:] # get all args after "--" main(argv) ```

yonghao lv commented

2022-05-03 13:59:22 +02:00

Added subscriber: @1029910278

Philipp Oeser commented

2022-05-03 15:08:01 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2022-05-03 15:08:01 +02:00

Not sure about your script, but the other issue is either #70584 or #96023

yonghao lv commented

2022-05-03 15:48:27 +02:00

Yeah, I think it is similar.
But I check the console, there is a zero divide error when executing SIZE_Y = floor(SIZE_X / (img.size- [x] / img.size[1]))
I am not sure why getting size from large image will cause this problem

Yeah, I think it is similar. But I check the console, there is a zero divide error when executing `SIZE_Y = floor(SIZE_X / (img.size- [x] / img.size[1]))` I am not sure why getting size from large image will cause this problem

Harley Acheson commented

2022-05-04 01:43:15 +02:00

Added subscriber: @Harley

Harley Acheson commented

2022-05-04 01:43:15 +02:00

I expect that the problem is less about the sizes of the individual files and more about the number of them in the folder. That "Pro Studios Metal" folder typically holds about 90 EXR files and our current code is not very good at thumbnailing that many. They are done quite a few at a time and are expanded to full-size uncompressed float buffers, so you can exhaust available memory as shown in #70584 (Heavy EXR images use too much memory when loading them (e.g. multi-threaded thumbnail generation causes swapping)) and #96023 (Blender File Browser not generating previews for large (filezise) images)

I have a patch that might one day help with this: D14663: IMBUF: Thumbnails of all EXR files using less RAM, but in the mean time the best advice is to put less of them in each folder.

I expect that the problem is less about the sizes of the individual files and more about the number of them in the folder. That "Pro Studios Metal" folder typically holds about 90 EXR files and our current code is not very good at thumbnailing that many. They are done quite a few at a time and are expanded to full-size uncompressed float buffers, so you can exhaust available memory as shown in #70584 (Heavy EXR images use too much memory when loading them (e.g. multi-threaded thumbnail generation causes swapping)) and #96023 (Blender File Browser not generating previews for large (filezise) images) I have a patch that *might* one day help with this: [D14663: IMBUF: Thumbnails of all EXR files using less RAM](https://archive.blender.org/developer/D14663), but in the mean time the best advice is to put less of them in each folder.

yonghao lv commented

2022-05-10 22:56:59 +02:00

There are only 45 exr files in that folder.
20 of them is less than 15M, 6 of them is bigger than 20m (1 image >30m)
And I have 64gb memory, 50gb left

There are only 45 exr files in that folder. 20 of them is less than 15M, 6 of them is bigger than 20m (1 image >30m) And I have 64gb memory, 50gb left ![image.png](https://archive.blender.org/developer/F13065991/image.png) ![image.png](https://archive.blender.org/developer/F13065997/image.png)

Harley Acheson commented

2022-05-10 23:25:55 +02:00

But do all the images that are NOT thumbnailed still display correctly in the Blender Image Editor? Generally the current code does not properly handle Multiview, YCC, Spectral, and some other types of images. Images with a single RGB channel, like just Green, will show as Monochrome. There is a fairly long list of things we fail on, mostly in how we interpret channel names.

If you are able to compile source, give my patch a try - https://developer.blender.org/D14663 - and see if it thumbnails more of them. Just keep in mind that you'd have to clear your local thumbnail cache between tests (c:\users(username).thumbnails\large\ and C:\Users\Harley.thumbnails\fail\blender\ )

But do all the images that are NOT thumbnailed still display correctly in the Blender Image Editor? Generally the current code does not properly handle Multiview, YCC, Spectral, and some other types of images. Images with a single RGB channel, like just Green, will show as Monochrome. There is a fairly long list of things we fail on, mostly in how we interpret channel names. If you are able to compile source, give my patch a try - https://developer.blender.org/D14663 - and see if it thumbnails more of them. Just keep in mind that you'd have to clear your local thumbnail cache between tests (c:\users\(username)\.thumbnails\large\ and C:\Users\Harley\.thumbnails\fail\blender\ )

Omar Emara commented

2022-05-11 14:17:49 +02:00

Added subscriber: @OmarEmaraDev

Omar Emara commented

2022-05-11 14:17:49 +02:00

I assume Harley meant D14663 above not D1466, just in case you missed that.

I assume Harley meant [D14663](https://archive.blender.org/developer/D14663) above not [D1466](https://archive.blender.org/developer/D1466), just in case you missed that.

Germano Cavalcante commented

2022-05-16 19:42:56 +02:00

Added subscriber: @mano-wii

Germano Cavalcante commented

2022-05-16 19:42:56 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Germano Cavalcante commented

2022-05-16 19:42:56 +02:00

This seems like a limitation in the code for creating thumbnails and there are plans to improve the situation with D14663: IMBUF: Thumbnails of all EXR files using less RAM.
So I'm taking a step forward and confirming it as a Known Issue (to get the report out of the untriaged queue).

This seems like a limitation in the code for creating thumbnails and there are plans to improve the situation with [D14663: IMBUF: Thumbnails of all EXR files using less RAM](https://archive.blender.org/developer/D14663). So I'm taking a step forward and confirming it as a `Known Issue` (to get the report out of the untriaged queue).

Harley Acheson commented

2022-05-20 03:21:41 +02:00

Changed status from 'Confirmed' to: 'Resolved'

Harley Acheson closed this issue

2022-05-20 03:21:41 +02:00

Harley Acheson self-assigned this 2022-05-20 03:21:41 +02:00

Harley Acheson commented

2022-05-20 03:21:41 +02:00

Resolved with commit f600a2aa6d as that will thumbnail all EXRs no matter what type or size while using less RAM

Resolved with commit f600a2aa6d as that will thumbnail all EXRs no matter what type or size while using less RAM

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Heavy Exr file will not generate previews in asset broswer/file broswer #97810