Maybe I'm missing something, but I could not find documentation on how to use group nodes in custom node trees. The simplest solution is probably to provide an example in the Python docs on the bpy.types.NodeCustomGroup site. Without documenting how this is supposed to be done, we'll likely get more reports like T76591.
@Sergey (randum) provided an initial script that can be used. It needs to be refined a bit more though.