Alembic export doesn't take 'Custom Properties', as alembic non-standard data.
Open, NormalPublic

Description

System Information
Opensuse Leap 42.1 Linux, ATI graphics card

Blender Version
Blender 2.78b

Short description of error
If i add an custom properties, manually or by python api. Alembic export doesn't take care of them (custom properties).
If the alembic file is reloading in blender the custom properties are disappeared. Same for an extra 3d software (maya , guerilla...)
By exemple all 'extra attr' in maya are exported in native alembic export.
In a robust pipeline the absence of this 'custom properties' is a really problems.

Exact steps for others to reproduce the error
-Select cube
-type : pby.context.object["MyCustomProps"] = "foo"
or add it manually in properties panel.
-file > export alembic
-file > import alembic

Details

Type
Bug
Bastien Montagne (mont29) triaged this task as Normal priority.

@Kévin Dietrich (kevindietrich) again not sure whether this is TODO or bug or something else, will let you handle it. :)

@colin (col-one) do you have an example Alembic file, produced by Maya, that shows how Maya does this? Perhaps we can implement this in a compatible way.

@Sybren A. Stüvel (sybren) just some notes if you wish to tackle this on a rainy day before I can:

  • on Alembic side I guess it would make use of the ArbGeomParams, I have a lot of Alembic archives from various software (maybe we could have a public repo for them? also for regression tests purposes), some of which containing arbitrary properties. Also there is D2324 that could tackle some of this.
  • Blender side is the tricky part. The original patch used to handle custom properties, IIRC only for exports, however it was far from ideal. Campbell suggested to have a single Alembic property-group where all these custom settings can be stored (eg, object["alembic"]), but we scratched it due to lack of workflow examples.

Nevertheless, although some examples files and example workflows would be appreciated, this is more TODO than bug, except maybe for the case where properties are erased when reloading an Abc archive.

I have a lot of Alembic archives from various software (maybe we could have a public repo for them? also for regression tests purposes),

Yes please, share them!

The most important thing in a plurial software pipeline is to 'bake' extra properties in the .abc so we can bake a rigid nomenclature and avoid the mistake renaming or special characters by artists, also through different soft we keep It safe.
An other common use of this extra is used in a pipeline with standalone render engine like Guerilla. The workflow is based on this extra attr which has name 'tags'. With this tags it becomes possible to do shading / lookdev in parallel of animation and rigging. Shading artist generat a rendergraph without mesh who is it merged in final rendering scene with final abc files (who have extra attr tags). This technic will become standard workflow, all standard software manage this extra attr in abc, except 3dsmax and blender (for the moment). It's always used in big production as Illumination Mac Guff, Disney...

@colin (col-one) As I wrote in February: do you have an example Alembic file, produced by Maya, that shows how Maya does this? We don't need more motivation about how important this feature is, we need documentation and examples so that we can actually implement it.

Sorry for late... my previous answer was more for Kevin, and pipe details.
Here a simple alembic with some extra attr from maya. 'AnotherAttr' 'GuerillaTags'.

If i can send you more complex alembic i will do it soon.

Thanks a lot.

Hi all !

Is there any news about this improvement ?

Thanks !

I would love... but i think not ={

Ok, now i'm sad :(
Thanks for the info @colin (col-one) ! :)

Such custom properties aren't supported by Blender. This means that this isn't a bug report, but a feature request. We may implement this at some point, but not sure when we have time to work on it. A few simple example files do help, so thanks @colin (col-one) for providing one.

@Sybren A. Stüvel (sybren) In my older message there is a simple alembic with non-standard data, it come from maya. Tell me if you want more exemples.

Sorry for late... my previous answer was more for Kevin, and pipe details.
Here a simple alembic with some extra attr from maya. 'AnotherAttr' 'GuerillaTags'.

If i can send you more complex alembic i will do it soon.

Thanks a lot.

I know, I should have written "thanks @colin (col-one) for already providing one".

@colin (col-one): you wrote "If i can send you more complex alembic i will do it soon" -- any progress on that?

Hello, after our discussions at the bconf I thought it wise to propose a little synthesis about the management of Custom Properties and Alembic.

Contrary to ticket title, blender’s Custom Properties can be handled by classic arbitrary alembic’s ArrayProperty, This properties are data with form [key: value] who can be write on differents abc archive components like transform and shape. Data on others components like vertex, face… seems can be handled by non-standard data.
During the import of an abc archive the software can interpret this arbitary ArrayProperty data and use its for automation or to set values dependents of this data.
This data don’t disturb importation from a procedure who doesn’t interprets them.
For example Guerilla software interpret some abc’s Property data like [“guerilla_tags”: “foo, bar”] during the import proc if a Property data is find on a transform or shape component Guerilla will set the tags properties with its coma splitted values foo, bar.

Here some example extract with abcls, from a maya .abc and there is Properties on transform and shape, foo_transform foo_shape:

Using Alembic 1.7.1 (built Nov  1 2017 13:25:15)
  file written by: Maya 2016.5 AbcExport v1.0
  using Alembic : Alembic 1.5.8 (built Dec 24 2015 17:28:19)
  written on : Wed Nov 01 13:56:27 2017
  user description : Exported from: .
  core type : Ogawa
...
/obj1:
AbcGeom_PolyMesh_v1                objShape1 {
                                    schema=AbcGeom_PolyMesh_v1
                                    schemaBaseType=AbcGeom_GeomBase_v1
                                    schemaObjTitle=AbcGeom_PolyMesh_v1:.geom
                                   }
CompoundProperty                   .xform {schema=AbcGeom_Xform_v3}
/obj1/.xform:
CompoundProperty                   .arbGeomParams
CompoundProperty                   .userProperties
/obj1/.arbGeomParams:
ArrayProperty       string         foo_transform[1] {
                                    arrayExtent=1
                                    isGeomParam=true
                                    podExtent=1
                                    podName=string
                                   }
bar_transform
...
/obj2/objShape2/.arbGeomParams:
ArrayProperty       string         foo_shape[1] {
                                    arrayExtent=1
                                    isGeomParam=true
                                    podExtent=1
                                    podName=string
                                   }
bar_shape
...
/obj3:
AbcGeom_PolyMesh_v1                objShape3 {
                                    schema=AbcGeom_PolyMesh_v1
                                    schemaBaseType=AbcGeom_GeomBase_v1
                                    schemaObjTitle=AbcGeom_PolyMesh_v1:.geom
                                   }
CompoundProperty                   .xform {schema=AbcGeom_Xform_v3}
/obj3/.xform:
CompoundProperty                   .arbGeomParams
CompoundProperty                   .userProperties
/obj3/.arbGeomParams:
ArrayProperty       string         foo_transform[1] {
                                    arrayExtent=1
                                    isGeomParam=true
                                    podExtent=1
                                    podName=string
                                   }
bar_transform
...
/obj3/objShape3/.arbGeomParams:
ArrayProperty       string         foo_shape[1] {
                                    arrayExtent=1
                                    isGeomParam=true
                                    podExtent=1
                                    podName=string
                                   }
bar_shape
...

In blender how could this work ?

Export is the easy way, there is no interpretations questions about this abc's Properties.
Firstable the arbitrary ArrayProperties can be defined by the blender’s Custom Properties present inside object or mesh properties because it’s easy to create them manually or with python.

Then, they will be written as arbitrary data on transform or shape abc archive component depend where the custom properties is (object or mesh).

How decided which Custom Properties must be write in abc archive or not ?
The Maya way is maybe the best, inside the export abc panel it’s possible to precise which key name must be exported and/or which prefix key name (don’t have maya for a screen capture...). Also in the proc :

AbcExport -root |myChar -file test.abc -attr foo foo_py -attrprefix cycles foo

When AbcExport travers all objects from root myChar each time who it find foo, foo_py maya extra attr (blender’s Custom Properties equivalent) it will write them as ArrayProperties data.
So, in blender abc export ops can be defined with one or two extra argument, name_custom_properties and prefix_custom_properties.

For the future it might be possible to extrapolate this to the new blender’s collections system where defined collection’s Custom Properties will be propagate as arbitrary data on each object inside this collection during the abc writing.

For import it’s will be more complex, because each keys arbitrary properties data must will be discussed on how interpret it. For example if a data "cycle_subdiv": 2 is find must we set the cycle subdivision value of the object at 2 ? Or maybe just create a blender’s Custom Properties "cycle_subdiv": 2 on this object ?
Maybe we can find a more generic solution that will leave the different blender’s domains to freely interpret these data.

Here the file text from abcls:


And here the abc archive:

For the moment, it is very difficult for me to find a complex alembic file that is not proprietary. But the complexity resides more in the hierarchical structure rather than in the writing of the properties who are present only on transformers and shapes.
But if i can catch one i will send you anyway.

Please consider this proposal, better the Alembic will be integrated into blender at the most blender will be used in serious production.

Thanks a lot !

I noticed that the custom properties are arrays of extent 1 (for example, "guerilla_tags": ["foo, bar"]. Do you have any idea why that would be? Why not simply write them as Scalar properties instead, i.e. "guearilla_tags: "foo, bar"?

These properties are defined on .arbGeomParams; does all software write its custom properties there? There is also .userProperties, which we currently use to write some data about stereo distance and eye separation for multi-view cameras. Looking at the name that would seem like a more logical spot to place custom properties. Would that be understood by Maya, Houdini, etc. too?

Furthermore, do these properties change over time? In other words, can they be animated or not?

I noticed that the custom properties are arrays of extent 1 (for example, "guerilla_tags": ["foo, bar"]. Do you have any idea why that would be? Why not simply write them as Scalar properties instead, i.e. "guearilla_tags: "foo, bar"?

Indeed we have already asked ourselves this question but I don't realy know, I don't know if it's a Maya export restriction or if it's alembic who owns just this type of container, will ScalarProperty exist?

colin (col-one) added a comment.EditedThu, Nov 2, 12:32 AM

These properties are defined on .arbGeomParams; does all software write its custom properties there? There is also .userProperties, which we currently use to write some data about stereo distance and eye separation for multi-view cameras. Looking at the name that would seem like a more logical spot to place custom properties. Would that be understood by Maya, Houdini, etc. too?

Humm, for the moment my knowledge is limited there, I will try to do some research on other software like 3dsmax, houdini... I pretty sure Arnold uses only on the .arbGeomParams. For .userProperties I never try, can you send me an archive with this kind of properties ? i will test.

Furthermore, do these properties change over time? In other words, can they be animated or not?

Yes it's a possibility, i.e: a custom property who manage texture animation in a rigging, who it will use later in the render.

I don't know if it's a Maya export restriction or if it's alembic who owns just this type of container, will ScalarProperty exist?

Yes, scalar properties exist, which is why I find this scalar-like array approach a bit strange.

Humm, for the moment my knowledge is limited there, I will try to do some research on other software like 3dsmax, houdini... I pretty sure Arnold uses only on the .arbGeomParams.

Looking forward to your findings!

For .userProperties I never try, can you send me an archive with this kind of properties ? i will test.

I'll post a file here once I've implemented writing of custom properties. Then it should be easy to write to either .abcGeomParams or .userProperties and see what is supported by which software.

Yes it's a possibility, i.e: a custom property who manage texture animation in a rigging, who it will use later in the render.

Ok.

Ok, here an abcls from houdini alembic with a string attribute foo_houdini

CompoundProperty                   .arbGeomParams
/sphere_object1/attribcreate1/.arbGeomParams:
ArrayProperty       string         foo_houdini[1] {
                                    arrayExtent=1
                                    geoScope=con
                                    isGeomParam=true
                                    podExtent=1
                                    podName=string
                                   }
bar_houdini

Here the file :

Like maya houdini write string in a ArrayProperty... and in .arbGeomParams , maybe it's a conventional accord. I ask somes question at guerilla render developers about management of this attribute.

In houdini, if I change the type of the attribute, for example to int, it continues to write in ArrayProperty of type int.
With houdini it is possible to put several values in an attribute and now we can see the arrayExtent envolve.

ArrayProperty       int32_t        foo_h_int2[1] {
                                    arrayExtent=2
                                    geoScope=con
                                    isGeomParam=true
                                    podExtent=1
                                    podName=int32_t
                                   }
777
666

ArrayProperty       int32_t        foo_h_int3[1] {
                                    arrayExtent=3
                                    geoScope=con
                                    isGeomParam=true
                                    podExtent=1
                                    podName=int32_t
                                   }
777
666
555

ArrayProperty       int32_t        foo_h_int4[1] {
                                    arrayExtent=4
                                    geoScope=con
                                    isGeomParam=true
                                    podExtent=1
                                    podName=int32_t
                                   }
777
666
555
444

News from Guerilla render's dev :
Guerilla reads similary way .arbGeomParams and .userProperties.

I don't know about Arnold or Renderman ....

Thanks for the info, this is very useful. I have implemented reading support for those properties in my private branch, albeit without animation support and not yet for all object types (currently at object itself and mesh data, and only in .arbGeomParams).

Could you maybe create a file for me that has a few properties animated? It would be great if you could include int/float singleton, int/float array, and string properties.

Here some alembic from Houdini, attribute foo animated on 10 frames int float and string. It's look like weird, I dont realy understand how animation work in alembic. I hope it will be useful !