the new Keying node has a problem with colored edges after pulling a matte from a "perfect" greenscreen.
The attached file shows the problem.
I believe this can be done better, I will attach a patch soon.
till then, David.
Here is a patch for the problem.
A couple of notes:
This works better because the factor fabsf(1.0f - val) is taken out of the saturation
equation, making it linear. This creates a different problem, unsaturated colors are not
keyed anymore, this is fixed by adding a maximum norm luminance key after the
saturation key. This should also make the overexposure check redundant. Lastly, the
if block is replaced by a CLAMP macro, making it much shorter.
I have done quite a lot of tests of this method, and I feel that keyers in other software
work similarly (especially concerning the linear nature of the key, resulting in clean
mattes in perfect conditions).
Regarding the patch, I suggest not using macros this way, because this gets expand and the math operations will be done more times. Also means the comments are lost.
Probably this patch needs some good tests on Mango shots, I know Sergey experimented with many formulas so there might be a reason this code is here.
Thanks for looking at the patch. I attached a new version that fixes the issue with the macros.
In my opinion the comments are not really that useful, as the first one does not apply anymore,
and the others basically say "clamp to 0-1 range".
I tested this with two of the mango shots, and various other shots in my own projects and I
feel the results are consistently better, especially when combined with a screen subtraction
(hint at my other patch: http://projects.blender.org/tracker/index.php?func=detail&aid=32797&group_id=9&atid=127 ).
Checked the patch. It behaves better with the scene you've attached, and behaves pretty much the same on mango files. However, I wouldn't consider it behaves much better in plates from holllywood camera works. You could check it on "godiva medium" and "simple shadow extraction" plates from http://www.hollywoodcamerawork.us/greenscreenplates.html i.e.
From what i understand, you can not actually have single formula which would always performs good, perhaps it should be an option which formula to use.
Still not sure its actually a bug, just some case where current formula doesn't work well. I would prefer to have new formula as an option and also would ask to move to the bug tracker for further discussion.