OpenEXR DWA compression should not apply to non-color data passes #51231

New Issue

- · 2017-04-16T10:55:27+02:00

- commented

2017-04-16 10:55:27 +02:00

Linux Mint 17.3

Blender Version
2.78c

I write you because i've been testing alot with DWAA and DWAB. But there are major features missing. First of all. Blender need a Compression Tab kind of Setup button. I dont find anything in the release build 2.78c.
I uploaded a screenshot from the compositing app Natron to explain what i mean.

Second important thing is to make sure which passes get compressed and which are not. So it would be a good idea to create checkerboxes on each layer with a boolean description of "i want to compress this layer or not"

Layers with Capital letters not gonna get compressed. please read more in the foundry forum about that topic.: http://community.foundry.com/discuss/topic/125476

Cheers.

Linux Mint 17.3 **Blender Version** 2.78c I write you because i've been testing alot with DWAA and DWAB. But there are major features missing. First of all. Blender need a Compression Tab kind of Setup button. I dont find anything in the release build 2.78c. I uploaded a screenshot from the compositing app Natron to explain what i mean. ![Bildschirmfoto vom 2017-04-16 10:52:40.png](https://archive.blender.org/developer/F546927/Bildschirmfoto_vom_2017-04-16_10_52_40.png) Second important thing is to make sure which passes get compressed and which are not. So it would be a good idea to create checkerboxes on each layer with a boolean description of "i want to compress this layer or not" Layers with Capital letters not gonna get compressed. please read more in the foundry forum about that topic.: http://community.foundry.com/discuss/topic/125476 Cheers.

- commented

2017-04-16 10:55:27 +02:00

Changed status to: 'Open'

- commented

2017-04-16 10:55:27 +02:00

Added subscriber: @AlexanderWeide

Aaron Carlisle commented

2017-04-16 18:30:02 +02:00

Added subscriber: @Blendify

Aaron Carlisle commented

2017-04-16 18:30:02 +02:00

Changed status from 'Open' to: 'Archived'

Aaron Carlisle closed this issue

2017-04-16 18:30:02 +02:00

Aaron Carlisle self-assigned this 2017-04-16 18:30:02 +02:00

Aaron Carlisle commented

2017-04-16 18:30:02 +02:00

Hi, while we love to hear these ideas, this is not the right place for them.
This website is mainly used to track bug reports and not feature requests.
But luckly, there are some other nice solutions you can take,
so please use one of the other forms of communication listed below:

https://rightclickselect.com/
Fun board mailing list: https://lists.blender.org/mailman/listinfo/bf-funboard

Hi, while we love to hear these ideas, this is not the right place for them. This website is mainly used to track bug reports and not feature requests. But luckly, there are some other nice solutions you can take, so please use one of the other forms of communication listed below: - https://rightclickselect.com/ - Fun board mailing list: https://lists.blender.org/mailman/listinfo/bf-funboard

- commented

2017-04-16 18:34:30 +02:00

Hmm, Not nice to hear.. I think you dont get it.. ITs not a missing feature or a nice to have, its essential. The entire DWAA and DWAB Support is useless... because with the actuall methode you compressing all Renderlayers.. like ZDeph and other Aux channels. By doing that you entirely kill the Renderlayer...Its worthless to use it. I wrote it here because you need to Fix this ASAP, otherwise no serious Artist should or will use it. ..

Best regards.

Hmm, Not nice to hear.. I think you dont get it.. ITs not a missing feature or a nice to have, its essential. The entire DWAA and DWAB Support is useless... because with the actuall methode you compressing all Renderlayers.. like ZDeph and other Aux channels. By doing that you entirely kill the Renderlayer...Its worthless to use it. I wrote it here because you need to Fix this ASAP, otherwise no serious Artist should or will use it. .. Best regards.

Aaron Carlisle commented

2017-04-16 18:41:22 +02:00

Hey, I think we already support DWAA and DWAB EXR compression? So I am a little confused about your request.

- commented

2017-04-16 18:49:21 +02:00

Hi, yes thats true, but, you dont brought it to the end. You implemented 50% of the DWAA and DWAB support. I can't see any Compression Rate Tab or Setting, but that is really crucial. At which Rate is your DWAA or DWAB compressing? Propper Values are between 20 and 500. A DWAA Layer is used as Compositing Layer with a Production Ready Compressing between 30 and 50. A final composed Layer will be saved with DWAB at 150.

The next important thing is, That blenders Compositor isn't able to read those files of Multilayer EXR's. In short words Blender is not able to read all Renderlayers in the Multilayer EXR DWAA/DWAB Compressed file.

So i want to help you that the responsible code designer understands that there are important missing features. The current version in 2.78c is useless in short words crap. So it would be a good idea to fix it ASAP so that larger productions can use it how it should behave in almost every other Comp/3d app.

To get it done please contact the Natron developer, he integrated it properly into Natron so hopefully he can help you. Please also read the third comment from Deke Kincaid
August 27, 2016 - 3:48 PM http://community.foundry.com/discuss/topic/125476 it will help you to understand problem.

Best Regards ;)

Hi, yes thats true, but, you dont brought it to the end. You implemented 50% of the DWAA and DWAB support. I can't see any Compression Rate Tab or Setting, but that is really crucial. At which Rate is your DWAA or DWAB compressing? Propper Values are between 20 and 500. A DWAA Layer is used as Compositing Layer with a Production Ready Compressing between 30 and 50. A final composed Layer will be saved with DWAB at 150. The next important thing is, That blenders Compositor isn't able to read those files of Multilayer EXR's. In short words Blender is not able to read all Renderlayers in the Multilayer EXR DWAA/DWAB Compressed file. So i want to help you that the responsible code designer understands that there are important missing features. The current version in 2.78c is useless in short words crap. So it would be a good idea to fix it ASAP so that larger productions can use it how it should behave in almost every other Comp/3d app. To get it done please contact the Natron developer, he integrated it properly into Natron so hopefully he can help you. Please also read the third comment from Deke Kincaid August 27, 2016 - 3:48 PM http://community.foundry.com/discuss/topic/125476 it will help you to understand problem. Best Regards ;)

Aaron Carlisle commented

2017-04-16 18:49:50 +02:00

Note that per render layer compression can be accomplished with the file output node in the compositor.
The only thing I think that might be lacking is the ability to do this for a multilayer EXR.

Note that per render layer compression can be accomplished with the file output node in the compositor. The only thing I think that might be lacking is the ability to do this for a multilayer EXR.

Troy Sobotka commented

2017-04-16 19:15:38 +02:00

Added subscriber: @troy_s

Troy Sobotka commented

2017-04-16 19:15:38 +02:00

Jesus Alex, you are at it again with your hyperbolic rubbish.

So it would be a good idea to fix it ASAP so that larger productions can use it how it should behave in almost every other Comp/3d app.

That's right, all of the larger productions with millions of dollars in their pipelines invested in Nuke are going to stop everything and grab Blender because now there is a compression ratio control exposed somewhere you weren't looking.

facepalm

Jesus Alex, you are at it again with your hyperbolic rubbish. > So it would be a good idea to fix it ASAP so that larger productions can use it how it should behave in almost every other Comp/3d app. That's right, all of the larger productions with millions of dollars in their pipelines invested in Nuke are going to stop everything and grab Blender because now there is a compression ratio control exposed somewhere you weren't looking. *facepalm*

Brecht Van Lommel changed title from ~~Important missing Feature Request for DWAA and DWAB.~~ to OpenEXR DWA compression should not apply to non-color data passes

2017-04-16 19:35:36 +02:00

Brecht Van Lommel commented

2017-04-16 19:35:36 +02:00

Changed status from 'Archived' to: 'Open'

Brecht Van Lommel reopened this issue

2017-04-16 19:35:36 +02:00

Brecht Van Lommel commented

2017-04-16 19:40:56 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2017-04-16 19:40:56 +02:00

I agree there is a bug here, renamed the title to reflect that. I don't think we need manual control, just for some passes with vector/ID/.. types instead of color we need to tell OpenEXR not to apply lossy compression. The same goes for half float compression really, that's also not suitable for most data passes.

Aaron Carlisle was unassigned by Brecht Van Lommel

2017-04-16 19:42:01 +02:00

- commented

2017-04-16 19:46:11 +02:00

Ok accepted. At which rate is dwaa and dwab compressing? I asked because if you maybe scaleup the pass from 2k to 4k it would be nice when the compression rate is not above 50. Thats why i asked about a manual slider for compression. I guess if it would compress at 45 as default. all should be fine. Thanks for reactivating the issue and yeah happy easter.

Brecht Van Lommel commented

2017-04-16 20:53:58 +02:00

Blender uses the default compression level of 45.

So OpenEXR applies lossy compression only to the following channel names: R, G, B, Y, BY, RY. The issue here is the Y channel, which can not only be luminance but also the Y in XYZ.

This appears to be hardcoded, no way to change this without editing the OpenEXR source code. The OpenEXR docs contain channel name conventions for various cases, but not for XYZ vector coordinates.

A workaround is to use lowercase xyz channel names. However Blender itself assumes upper case channel names when reading files, so needs to become case insensitive. The question is how many other applications will not be able to read such files with lower case channel channels. From a quick look Natron also seems like it assumes upper case XYZ channel names.

Blender uses the default compression level of 45. So OpenEXR applies lossy compression only to the following channel names: R, G, B, Y, BY, RY. The issue here is the Y channel, which can not only be luminance but also the Y in XYZ. This appears to be hardcoded, no way to change this without editing the OpenEXR source code. The [OpenEXR docs ](http://www.openexr.com/TechnicalIntroduction.pdf) contain channel name conventions for various cases, but not for XYZ vector coordinates. A workaround is to use lowercase xyz channel names. However Blender itself assumes upper case channel names when reading files, so needs to become case insensitive. The question is how many other applications will not be able to read such files with lower case channel channels. From a quick look [Natron ](https://github.com/MrKepzie/openfx-io/blob/master/OIIO/ReadOIIO.cpp) also seems like it assumes upper case XYZ channel names.

Brecht Van Lommel commented

2017-04-16 20:58:41 +02:00

Perhaps the best solution here would not be to change anything in Blender, but to fix OpenEXR to only apply lossy compression to the Y channel when it is the only channel or if it goes along with BY and RY channels.

Arnaud Couturier commented

2017-10-17 06:55:31 +02:00

Added subscriber: @piiichan

Arnaud Couturier commented

2017-10-17 06:55:31 +02:00

Hello, not sure where to find someone who could answer that specific question, but in Blender 2.79, OpenEXR DWAB compression has been removed, there's only DWAA left. Why is this?
Isn't DWAB supposed to be slightly better compressed and generally faster to load?

Hello, not sure where to find someone who could answer that specific question, but in Blender 2.79, OpenEXR DWAB compression has been removed, there's only DWAA left. Why is this? Isn't DWAB supposed to be slightly better compressed and generally faster to load?

- commented

2017-10-17 10:53:59 +02:00

In #51231#466043, @piiichan wrote:
Hello, not sure where to find someone who could answer that specific question, but in Blender 2.79, OpenEXR DWAB compression has been removed, there's only DWAA left. Why is this?
Isn't DWAB supposed to be slightly better compressed and generally faster to load?

Its really missing yes. And important. A other problem with 2.79 is that Blender 2.79 on Linux is not able anymore to read EXR File textures without problems. Blender is slowing down massivly and consume almost the entire CPU when you try to read exr textures. I can confirm that this is only a problem in Blender 2.79. This also need to be fixed soon.

> In #51231#466043, @piiichan wrote: > Hello, not sure where to find someone who could answer that specific question, but in Blender 2.79, OpenEXR DWAB compression has been removed, there's only DWAA left. Why is this? > Isn't DWAB supposed to be slightly better compressed and generally faster to load? Its really missing yes. And important. A other problem with 2.79 is that Blender 2.79 on Linux is not able anymore to read EXR File textures without problems. Blender is slowing down massivly and consume almost the entire CPU when you try to read exr textures. I can confirm that this is only a problem in Blender 2.79. This also need to be fixed soon.

Steffen Dünner commented

2017-10-17 11:56:49 +02:00

Added subscriber: @SteffenD

Brecht Van Lommel commented

2017-10-17 12:02:48 +02:00

DWAB was disabled due to #50673.

If there is a problem reading EXR files, please report a bug with example file to reproduce, because I don't see any problems in my tests.

DWAB was disabled due to #50673. If there is a problem reading EXR files, please report a bug with example file to reproduce, because I don't see any problems in my tests.

Arnaud Couturier commented

2017-10-17 14:19:37 +02:00

Thank you Brecht, I couldn't find an answer, now I have it :)
I hope this can be solved sometime down the road.
I don't have any problem on my side though, with reading and writing EXR files with Blender.

Thank you Brecht, I couldn't find an answer, now I have it :) I hope this can be solved sometime down the road. I don't have any problem on my side though, with reading and writing EXR files with Blender.

Lsscpp commented

2019-02-05 01:02:08 +01:00

Added subscriber: @lsscpp

Lsscpp commented

2019-02-05 01:02:08 +01:00

Is there a reason why compression factor isn't exposed to the user? link

Is there a reason why compression factor isn't exposed to the user? [link ](https://blender.community/c/rightclickselect/D1bbbc/)

Jose commented

2019-09-06 13:44:04 +02:00

Added subscriber: @Dogway

Jose commented

2019-09-06 13:52:07 +02:00

It should expose the quality slider in the same way it is exposed for JPGs.
If the Open EXR specification allows it it should also allow for compression of other layers/channels.

PIZ - If aiming for quality (lossless) (default for archiving Digital EXR masters)
DWAA - If aiming to save space (lossy but transparent compression, might be good for albedos)
RLE - Good for large uniform areas like utility maps (lossless)
PXR24 - Compressing High Dynamic Range passes (Depth, alphas) (lossy)

It should expose the quality slider in the same way it is exposed for JPGs. If the Open EXR specification allows it it should also allow for compression of other layers/channels. > PIZ - If aiming for quality (lossless) (default for archiving Digital EXR masters) > DWAA - If aiming to save space (lossy but transparent compression, might be good for albedos) > RLE - Good for large uniform areas like utility maps (lossless) > PXR24 - Compressing High Dynamic Range passes (Depth, alphas) (lossy)

Lsscpp commented

2019-09-21 11:56:29 +02:00

Dwaa compression has a different value range, it's not 0/100%, but anyway, a slider would be appreciated

Philipp Oeser commented

2020-02-10 14:14:05 +01:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2020-02-10 14:14:05 +01:00

Hm, this still being an issue, but one that has no apparent/obvious solution yet (talking about the Y in XYZ), I am kind of unsure how to classify this report...

I stumbled about this comment (from here )

Yes, we are aware of the Y vs y issue; it is possible to tell the DWA compressor to use different rules and leave the Y channel uncompressed and I hope we can look into it soon.

Or the comments here :

If initializeDefaultChannelRules() doesn't quite suite your naming conventions, you can adjust the rules without breaking decoder compatability.

... but havent done any further investigations... (not sure if this can be done without modifier EXR source...)

Since this is kind of breaking (some) channels though, will classify as Bug for now... (@brecht: please correct me if I am wrong...)

Hm, this still being an issue, but one that has no apparent/obvious solution yet (talking about the `Y` in `XYZ`), I am kind of unsure how to classify this report... I stumbled about this comment (from [here ](https://forums.chaosgroup.com/forum/v-ray-for-maya-forums/v-ray-for-maya-general/76037-rendering-exr-with-dwaa-compression)) > Yes, we are aware of the Y vs y issue; it is possible to tell the DWA compressor to use different rules and leave the Y channel uncompressed and I hope we can look into it soon. Or the comments [here ](https://github.com/AcademySoftwareFoundation/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp): > If **initializeDefaultChannelRules()** doesn't quite suite your naming conventions, you can adjust the rules without breaking decoder compatability. ... but havent done any further investigations... (not sure if this can be done without modifier EXR source...) Since this is kind of breaking (some) channels though, will classify as Bug for now... (@brecht: please correct me if I am wrong...)

Christian Beckers commented

2020-05-03 15:47:34 +02:00

Added subscriber: @BeckersC

Mel Massadian commented

2023-01-07 16:33:16 +01:00

Added subscriber: @mel

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

OpenEXR DWA compression should not apply to non-color data passes #51231