Cycles: Proof-of-Concept Implementation of Cryptomatte Matte ID pass system
Needs RevisionPublic

Authored by Lukas Stockner (lukasstockner97) on Jul 15 2016, 11:42 PM.

Details

Reviewers
Brecht Van Lommel (brecht)
Group Reviewers
Cycles
Summary

This patch implements the necessary bits for supporting the Cryptomatte system
(https://github.com/Psyop/Cryptomatte) which is designed to allow efficient generation
of arbitrary ID matte passes.
To do so, the code doesn't just store the Object and Material ID of the first sample as usual,
but instead stores multiple IDs, each with an associated weight. In the code, this is currently
implemented with 16 slots, but more could be added by changing a define.
For the current Object and Material ID passes, the code then uses the ID with the highest weight,
which already avoids the noisy boundaries that currently appear around out-of-focus objects.
The actual Cryptomatte passes contain the first few IDs and their weight, sorted by weight.
Therefore, the first pass contains the same IDs as the classic passes.

A few bits are still missing for full Cryptomatte support:

  • Their specification includes generating the IDs by hashing names and storing the name->ID mapping as metadata in the output EXRs. This patch just uses the current Cycles pass IDs.
  • For a final version, these passes must be exposed. Right now, however, adding new passes is tricky. Also, if the user is to be able to choose the number of passes, a refactored pass system is required. For now, the first six Object IDs are stored in Diffuse/Glossy Direct/Indirect/Color, while the Material IDs are stored in Transmission/Subsurface D/I/C.
  • The official Cryptomatte passes contain two ID/weight sets each, one in the R and G passes and one in B/A. Since the temporary passes used in this patch only have 3 components, each of them just contains one set and B is zero.

Also, if this is to be supported officially, a compositor node to extract a black-white pass for a specific ID would of course
be needed as well.

Diff Detail

Repository
rB Blender
Branch
standalone_cryptomatte (branched from master)
Build Status
Buildable 340
Build 340: arc lint + arc unit
Lukas Stockner (lukasstockner97) retitled this revision from to Cycles: Proof-of-Concept Implementation of Cryptomatte Matte ID pass system.Jul 15 2016, 11:42 PM

Also, P375 contains a patch that adds a compositor node for extracting masks for individual IDs.

@Lukas Stockner (lukasstockner97) Just looked up what a Cryptomatte is and this would be a killer feature!
@Sean Kennedy (hype) You don't mind, if I add you as a subscriber? I believe this is something for you too. :-)

Thanks for the add! This sounds great!

Elegantly solved problem for a really needed feature indeed :)

Brecht Van Lommel (brecht) requested changes to this revision.EditedJul 31 2016, 5:41 PM

Better masking would be great. There's basically two fairly independent parts to Cryptomatte:

  • A system to generate unique ID's per object automatically and add an ID <-> name mapping that can be used at the compositing stage. This can work with e.g. the existing object ID pass.
  • A way to generate antialiased mattes for all objects in the scenes. An alternative way to do this would be to use deep EXR, where you have the additional advantage of being able to extract an object cleanly. Cryptomatte only tells you which objects contribution to a given pixels and how much, but it doesn't tell you what they contributed. The main advantage is that it takes up less space and works without a deep compositing pipeline.

I think this is useful functionality, but since this is only prototype code I'm marking it with request changes.

This revision now requires changes to proceed.Jul 31 2016, 5:41 PM

This patch was mentioned in a Natron de thread here: https://forum.natron.fr/t/natron-3d-distributed-node-graph/1237/5

I was wondering if anyone would be willing to guide me into contributing some time to have this patch pulled in?

Thanks!

Vincent

Hi, great to see interest in that patch!

The code right here is just a quick test and would need to be completely redone for a proper integration.
Some points to consider:

  • A big part of the Cryptomatte specification deals with how exactly the Name<->ID mapping works and is stored in the EXR. Since Cycles (in Blender) and EXR writing are separated by quite a few layers of abstraction, it would take a lot of glue code to get that info into the EXR. The best way probably would be to have all the ID stuff happening in Blender, which would then expose a RNA parameter that gives Cycles the ID for every object.
  • Another issue is pass handling - in this patch, I replaced a few of Cycles' passes with the Cryptomatte passes, but that's of course impossible for the final patch. However, we're out of pass IDs currently, and even the hack that I use in my denoising branch to extend those is not enough since the user should be able to select a variable amount of passes. Therefore, we'll need to improve the pass system and get rid of the bitfield approach altogether first.
  • Probably not relevant for Natron, but if Cycles can generate these passes, it should be possible to visualize them in the Image Editor.

So, while this patch shows that it's technically possible to generate these passes from Cycles, there's a lot of work required on the Blender side of things in rather tricky places to get it working...

For some experiments, I could probably get it into Cycles standalone a lot faster.

Yes yippee yay, standalone is what I'm planning to use anyways (to integrate in Natron). Does it eliminate the "path to EXR" AND the "pass count limit" problems ?

Vincent

Okay, here's a rewritten and cleaner implementation into the Cycles kernel and render code.

Of course, the integration into Blender is basically missing completely, although it already improves results for DoF and MoBlur - but all the interfaces for integration into any host application should be there.

From your side, you'd need to:

  • Add PASS_MATERIAL_ID and/or PASS_OBJECT_ID to enable the ID passes
  • Set the pass IDs of the shaders and objects
  • Set object_id_slots and/or shader_id_slots in the BufferParams AND the Film (the 4 used here is just an example, in practise you'd probably want something like the number of slots you want to use in the end plus 3 or so).
  • Get the data using something like the get_matte_rect function (that's just an example, I didn't test it at all)

This patch is not ready for actual review yet as long as it's missing all the Blender-side changes, but for external integration it should be all you need :)

Thanks! I will try it out tomorrow. I'm on macOS 10.11.6 and I was not able to compile with yesterday's code cloned from here: git://git.blender.org/cycles.git

But I will look into it further and apply your patch and see how it goes.

This revision now requires changes to proceed.May 5 2017, 3:21 PM