FFmpeg unpremultiply 2.79 #52988

Closed
opened 2017-10-04 07:18:23 +02:00 by K.nao · 19 comments

System Information
WIn10 GTX980Ti

Blender Version
2.79bd8ac9

Short description of error
When OpenGL render, alpha processing when saving with FFmpeg differs between 2.78 and 2.79.
It is not Unpremultiply 2.79.

Exact steps for others to reproduce the error
Add Background images > Draw Front > Opacity to lower > OpenGL render Animation(FFmpeg Video/2.78 H.264) save
キャプチャ.JPG

**System Information** WIn10 GTX980Ti **Blender Version** 2.79bd8ac9 **Short description of error** When OpenGL render, alpha processing when saving with FFmpeg differs between 2.78 and 2.79. It is not Unpremultiply 2.79. **Exact steps for others to reproduce the error** Add Background images > Draw Front > Opacity to lower > OpenGL render Animation(FFmpeg Video/2.78 H.264) save ![キャプチャ.JPG](https://archive.blender.org/developer/F948478/キャプチャ.JPG)
Author

Changed status to: 'Open'

Changed status to: 'Open'
Author

Added subscriber: @KNAO

Added subscriber: @KNAO

#53543 was marked as duplicate of this issue

#53543 was marked as duplicate of this issue

Added subscriber: @Sergey

Added subscriber: @Sergey

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'
Sergey Sharybin self-assigned this 2017-10-04 11:49:12 +02:00

After studying own file about this issue came to conclusion that the current behavior is more correct. What was happening in 2.78 is:

  • OpenGL render was giving byte buffer with straight alpha in sRGB space
  • Render pipeline was converting this buffer to linear space with premultiplied alpha (since this is what render pipeline is supposed to deliver for regular renders).
  • Saving and drawing pipeline was converting linear premultipled buffer back to sRGB with straight alpha.

There is no correct way to apply color space conversion on a buffer with alpha, so our approximation was causing some brightening on colors.

What happens in 2.79 is:

  • OpenGL render was giving byte buffer with straight alpha in sRGB space
  • Render pipeline stores it as a byte buffer, with straight alpha. Similar to output of VSE. No color space conversion is involved.
  • Saving and drawing pipeline does not do any conversions either.

This was done as an optimization (to avoid byte->float->byte roundtrip), and since there is no color space conversion and alpha transform involved, there is no brightening involved either.

So while new behavior is different from 2.78, it should match exactly to what you see in viewport, what you see in OpenGL render result and what you see in the generated file. If that's not the case, please attach demo .blend file which demonstrates the issue.

After studying own file about this issue came to conclusion that the current behavior is more correct. What was happening in 2.78 is: - OpenGL render was giving byte buffer with straight alpha in sRGB space - Render pipeline was converting this buffer to linear space with premultiplied alpha (since this is what render pipeline is supposed to deliver for regular renders). - Saving and drawing pipeline was converting linear premultipled buffer back to sRGB with straight alpha. There is no correct way to apply color space conversion on a buffer with alpha, so our approximation was causing some brightening on colors. What happens in 2.79 is: - OpenGL render was giving byte buffer with straight alpha in sRGB space - Render pipeline stores it as a byte buffer, with straight alpha. Similar to output of VSE. No color space conversion is involved. - Saving and drawing pipeline does not do any conversions either. This was done as an optimization (to avoid byte->float->byte roundtrip), and since there is no color space conversion and alpha transform involved, there is no brightening involved either. So while new behavior is different from 2.78, it should match exactly to what you see in viewport, what you see in OpenGL render result and what you see in the generated file. If that's not the case, please attach demo .blend file which demonstrates the issue.
Author

I tested it again.

I am sorry It seems to be a problem of the preservation process rather than the problem of OpenGL render.

I tried to save it with RGB and RGBA with PNG easy to understand.
As with FFmpeg, alpha processing is not appropriate when saving RGB.

RGBA save
RGBA_save.png

RGB save
RGB_save.png

RGBA save>Nuke>RGB save(This is the best)
RGBA_save-NukeRGB_save.png

Screen shot
ScreenShot.JPG

test.blend

I tested it again. I am sorry It seems to be a problem of the preservation process rather than the problem of OpenGL render. I tried to save it with RGB and RGBA with PNG easy to understand. As with FFmpeg, alpha processing is not appropriate when saving RGB. RGBA save ![RGBA_save.png](https://archive.blender.org/developer/F950413/RGBA_save.png) RGB save ![RGB_save.png](https://archive.blender.org/developer/F950411/RGB_save.png) RGBA save>Nuke>RGB save(This is the best) ![RGBA_save-NukeRGB_save.png](https://archive.blender.org/developer/F950412/RGBA_save-NukeRGB_save.png) Screen shot ![ScreenShot.JPG](https://archive.blender.org/developer/F950420/ScreenShot.JPG) [test.blend](https://archive.blender.org/developer/F950406/test.blend)

Changed status from 'Archived' to: 'Open'

Changed status from 'Archived' to: 'Open'

Added subscriber: @brecht

Added subscriber: @brecht

Ok, see what happens now.

What happens is: saving image with alpha channel will alpha-over it on top of black background. This was added in 47d6d18771, and the reasoning here is following: if buffer has premultiplied alpha, we can't just ignore alpha channel, since that would cause colors to burn in areas where alpha is less than 1.

However, for unpremultiplied alpha images (such as OpenGL render) it's not really needed to alpha-over on top of black, and perhaps simple ignore of alpha channel is what we should do. Here is a quick patch for that:

P535: Fix #52988

diff --git a/source/blender/imbuf/intern/colormanagement.c b/source/blender/imbuf/intern/colormanagement.c
index 863bce125c5..ad04378cb66 100644
--- a/source/blender/imbuf/intern/colormanagement.c
+++ b/source/blender/imbuf/intern/colormanagement.c
@@ -1984,9 +1984,11 @@ ImBuf *IMB_colormanagement_imbuf_for_write(ImBuf *ibuf, bool save_as_render, boo
 {
 	ImBuf *colormanaged_ibuf = ibuf;
 	bool do_colormanagement;
-	bool is_movie = BKE_imtype_is_movie(image_format_data->imtype);
-	bool requires_linear_float = BKE_imtype_requires_linear_float(image_format_data->imtype);
-	bool do_alpha_under = image_format_data->planes != R_IMF_PLANES_RGBA;
+	const bool is_movie = BKE_imtype_is_movie(image_format_data->imtype);
+	const bool requires_linear_float = BKE_imtype_requires_linear_float(image_format_data->imtype);
+	const bool do_alpha_under = (ibuf->rect_float != NULL)
+	                                ? (image_format_data->planes != R_IMF_PLANES_RGBA)
+	                                : false;
 
 	if (ibuf->rect_float && ibuf->rect &&
 	    (ibuf->userflags & (IB_DISPLAY_BUFFER_INVALID | IB_RECT_INVALID)) != 0)

@brecht, wouldn't mind having second opinion here.

Ok, see what happens now. What happens is: saving image with alpha channel will alpha-over it on top of black background. This was added in 47d6d18771, and the reasoning here is following: if buffer has premultiplied alpha, we can't just ignore alpha channel, since that would cause colors to burn in areas where alpha is less than 1. However, for unpremultiplied alpha images (such as OpenGL render) it's not really needed to alpha-over on top of black, and perhaps simple ignore of alpha channel is what we should do. Here is a quick patch for that: [P535: Fix #52988](https://archive.blender.org/developer/P535.txt) ``` diff --git a/source/blender/imbuf/intern/colormanagement.c b/source/blender/imbuf/intern/colormanagement.c index 863bce125c5..ad04378cb66 100644 --- a/source/blender/imbuf/intern/colormanagement.c +++ b/source/blender/imbuf/intern/colormanagement.c @@ -1984,9 +1984,11 @@ ImBuf *IMB_colormanagement_imbuf_for_write(ImBuf *ibuf, bool save_as_render, boo { ImBuf *colormanaged_ibuf = ibuf; bool do_colormanagement; - bool is_movie = BKE_imtype_is_movie(image_format_data->imtype); - bool requires_linear_float = BKE_imtype_requires_linear_float(image_format_data->imtype); - bool do_alpha_under = image_format_data->planes != R_IMF_PLANES_RGBA; + const bool is_movie = BKE_imtype_is_movie(image_format_data->imtype); + const bool requires_linear_float = BKE_imtype_requires_linear_float(image_format_data->imtype); + const bool do_alpha_under = (ibuf->rect_float != NULL) + ? (image_format_data->planes != R_IMF_PLANES_RGBA) + : false; if (ibuf->rect_float && ibuf->rect && (ibuf->userflags & (IB_DISPLAY_BUFFER_INVALID | IB_RECT_INVALID)) != 0) ``` @brecht, wouldn't mind having second opinion here.

I think that fix makes sense. It's really a matter of different interpretations of alpha. With straight alpha you kind of assume that you have a full RGB image and then the alpha is a mask on top of that. In that case a user might expect to be able to recover the original image without the alpha mask. With premultiplied alpa images like renders it's not so much a mask that was applied after but rather there being no light there in the first place.

For straight alpha images, alpha-over on top of a black background is exactly the same operation as converting from straight to premultiplied and then dropping the alpha. For premultiplied float images, IMB_alpha_under_color_float() does not do much since it's already premultiplied, except setting alpha to 1 and setting the RGB to 0 when alpha is 0 (which I think is wrong actually).

I think that fix makes sense. It's really a matter of different interpretations of alpha. With straight alpha you kind of assume that you have a full RGB image and then the alpha is a mask on top of that. In that case a user might expect to be able to recover the original image without the alpha mask. With premultiplied alpa images like renders it's not so much a mask that was applied after but rather there being no light there in the first place. For straight alpha images, alpha-over on top of a black background is exactly the same operation as converting from straight to premultiplied and then dropping the alpha. For premultiplied float images, `IMB_alpha_under_color_float()` does not do much since it's already premultiplied, except setting alpha to 1 and setting the RGB to 0 when alpha is 0 (which I think is wrong actually).

@brecht, hrm actually after more testing i'm not sure what is really expected here.

For example, consider such input RGBA image:

rgba.png

Before this patch saving it as RGB will give you:

master.png

Which is quite expected i guess. At least that was an intention of the commit mentioned above. This is how Gimp will behave when you try to remove Alpha channel from RGBA image (it alpha-overs image onto a background color).

But with this patch you'll have:

patch.png

Which is quite useless i think.

Black is a bit arbitrary, so maybe alpha-over on white will be better alternative? Or.. a toggle whether to alpha-over or simply kick alpha out?

@brecht, hrm actually after more testing i'm not sure what is really expected here. For example, consider such input RGBA image: ![rgba.png](https://archive.blender.org/developer/F1041698/rgba.png) Before this patch saving it as RGB will give you: ![master.png](https://archive.blender.org/developer/F1041700/master.png) Which is quite expected i guess. At least that was an intention of the commit mentioned above. This is how Gimp will behave when you try to remove Alpha channel from RGBA image (it alpha-overs image onto a background color). But with this patch you'll have: ![patch.png](https://archive.blender.org/developer/F1041703/patch.png) Which is quite useless i think. Black is a bit arbitrary, so maybe alpha-over on white will be better alternative? Or.. a toggle whether to alpha-over or simply kick alpha out?

Honestly, I think leaving the behavior the same as it is now is quite acceptable. If you're going to render an OpenGL preview with a semi-transparent background image, you can't expect the result to be the same as if it was opaque. With Blender 2.8 and Eevee you'll be able to set up a world background image, and render out the animation with Eevee. Semi-transparent background images for viewport modelling are a separate thing then.

If you save an RGBA image as RGB, you need to composite it over some background color, and in general we can't guess which one the user wants. Black is as good a default as any in my opinion, white is going to look bad in other cases. We could add an option to choose the color, but maybe that's better left to the compositor.

Honestly, I think leaving the behavior the same as it is now is quite acceptable. If you're going to render an OpenGL preview with a semi-transparent background image, you can't expect the result to be the same as if it was opaque. With Blender 2.8 and Eevee you'll be able to set up a world background image, and render out the animation with Eevee. Semi-transparent background images for viewport modelling are a separate thing then. If you save an RGBA image as RGB, you need to composite it over some background color, and in general we can't guess which one the user wants. Black is as good a default as any in my opinion, white is going to look bad in other cases. We could add an option to choose the color, but maybe that's better left to the compositor.
Author

There is no problem saving RGBA.
But for RGB save alpha should be ignored.
"Channels of the image to draw: Color"
It is so with display switching. In the case of RGB Is something wrong with what I want to save with this?
キャプチャ.JPG

Also, even if you make the background Sky with OpenGL render, translucent alpha will be reflected, so even if you forcibly try to lose alpha it can not be avoided.
キャプチャ2.JPG

There is no problem saving RGBA. But for RGB save alpha should be ignored. "Channels of the image to draw: Color" It is so with display switching. In the case of RGB Is something wrong with what I want to save with this? ![キャプチャ.JPG](https://archive.blender.org/developer/F1134186/キャプチャ.JPG) Also, even if you make the background Sky with OpenGL render, translucent alpha will be reflected, so even if you forcibly try to lose alpha it can not be avoided. ![キャプチャ2.JPG](https://archive.blender.org/developer/F1134232/キャプチャ2.JPG)

Added subscribers: @davidmcsween, @chrisloughnane

Added subscribers: @davidmcsween, @chrisloughnane

When ffmpeg is first selected for output, there is no alpha channel available in output options.
In fact the color state boxes BW or RGB are not selected at all, the panel then defaults to RGBA (which isn't even an option button).

When the VSE is used to output this can result in an animation that uses alpha images over a VSE neutral background, the premultiply process is not applied.

I understand that no premult is desired (for alpha image sequence output), but I think that the default ffmpeg setting in 2.79 should at least be in RGB only mode.

When ffmpeg is first selected for output, there is no alpha channel available in output options. In fact the color state boxes BW or RGB are not selected at all, the panel then defaults to RGBA (which isn't even an option button). When the VSE is used to output this can result in an animation that uses alpha images over a VSE neutral background, the premultiply process is not applied. I understand that no premult is desired (for alpha image sequence output), but I think that the default ffmpeg setting in 2.79 should at least be in RGB only mode.

Changed status from 'Open' to: 'Archived'

Changed status from 'Open' to: 'Archived'

After extra thoughts i would consider this more a TODO. There is really no way to solve this issue when OpenGL operates in straight alpha mode. Any decision here will be arbitrary, and will solve one usecase but will ruing another one.

Proper solution would be to switch OpenGL to work with premultiplied alpha, which might still happen in 2.8 project.

After extra thoughts i would consider this more a TODO. There is really no way to solve this issue when OpenGL operates in straight alpha mode. Any decision here will be arbitrary, and will solve one usecase but will ruing another one. Proper solution would be to switch OpenGL to work with premultiplied alpha, which might still happen in 2.8 project.

Added subscriber: @1017965668

Added subscriber: @1017965668
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#52988
No description provided.