C++ / GLSL code reuse #103026

New Issue

Clément Foucault · 2022-12-08T14:31:44+01:00

Clément Foucault commented

2022-12-08 14:31:44 +01:00

With the latest developpement in the GPU module, it is now possible to inject C++ headers into GLSL shaders.

These headers most often have the _shader_shared.hh suffix. A nice example can be seen in eevee_next/eevee_shader_shared.hh . The content is always struct definitions (for data layout) and some static functions snippets.

File Type
The file is always written as C++ with some limitations. Only a subset of C++ is allowed so that source translation to GLSL is easier.
This choice is motivated to keep IDE working with all the definitions inside these shared headers. So the file extension is always .hh.

Source Translation
Many incompatible keywords are just #defined to null using preprocessor.
The GPUSource constructor then handles the 2 remaining cases that breaks the GLSL compilers:

enum expansion to const uint as GLSL do not support enum.
quote removal from #include directives as some compilers error out on any quote char present in non-comment code.
This preprocess is limited to shared header files to avoid the temptation to use C++ syntax in .glsl files.
See b42adab3a2 for more details.

Typenames
Using C++ typenames makes more sense in a C++ header. The GLSL translation layer define aliases to the GLSL types (float4 > vec4, float4x4 > mat4).
This was chosen because having things like using mat4 = float4x4 would make C++ code more complex and ambiguous to what type should be use in non shared files. The GLSL typenames are also quite ambiguous (ex: vec3, mat4, dvec2).

Math API
GLSL not being a Object Oriented language, we cannot rely on OOP function calls like my_vector.normalize() or my_vector.normalized(). Parsing and rewriting this syntax to functional syntax isn't trivial. For this reason, we are trying to stick to the functional syntax paradigm.
We might not want stick to GLSL naming convention for consistency reason (BLI invert() vs. GLSL inverse()). These cases are easily handled by some #define on GLSL side.
It is important to note that while the goal it to have a shared API, we might introduce some functionality only on the C++ side. Likewise, we can introduce some functions only on GLSL side if it makes sense.
For vectors: D13791: BLI: Refactor vector types & functions to use templates
For matrix: D16625: BLI: Refactor matrix types & functions to use templates

Padding
Vector types are still unaligned. Padding in struct definitions used in UBOs is still manual until we add similar alignment rules as GLSL.

float3 / (u)int3 still need manual padding and alignment to 16bytes.
float3x3 is prohibited in shared struct declarations as they "alias" to a float4x3 in GLSL.
This might need be improved in the future.

C++ subset
The most commonly missed feature for now are:

Parameter reference & (which would translate to inout in GLSL).
No arrays (syntax differs greatly in GLSL).
No pointers or references.
No iterators.
No namespace.
No template.
These are for shared GLSL/C++ code only. They are still allowed in the implementation of the API in C++ as long as the function call syntax can be emulated in GLSL.
This might need be improved in the future.

Adoption
This project was motivated by the EEVEE-Next project (#93220) and the whole viewport module is moving in a similar direction (Workbench, DrawManager, Overlays...). Other project such as the new sculpt paint engine (D16083) or geometry nodes have expressed their interest into moving computations to the GPU and this will help reduce friction.

Relevance:
It was brought up during the Metal Backend effort (#96261) that we might want to transition to HLSL for more friendly/modern shading language. This is currently out of scope as we still need to support OpenGL for a few more years until the vulkan backend takes over. After this, we might consider switching to HLSL and many of the issues highlighted above might be solved.

References:
https://docs.gl/sl4/acos
https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-intrinsic-functions
https://registry.khronos.org/OpenGL/specs/gl/GLSLangSpec.4.60.pdf

With the latest developpement in the GPU module, it is now possible to inject C++ headers into GLSL shaders. These headers most often have the `_shader_shared.hh` suffix. A nice example can be seen in [eevee_next/eevee_shader_shared.hh ](https://developer.blender.org/diffusion/B/browse/master/source/blender/draw/engines/eevee_next/eevee_shader_shared.hh). The content is always struct definitions (for data layout) and some static functions snippets. **File Type** The file is always written as C++ with some limitations. Only a subset of C++ is allowed so that source translation to GLSL is easier. This choice is motivated to keep IDE working with all the definitions inside these shared headers. So the file extension is always `.hh`. **Source Translation** Many incompatible keywords are just `#defined` to null using preprocessor. The `GPUSource` constructor then handles the 2 remaining cases that breaks the GLSL compilers: - `enum` expansion to `const uint` as GLSL do not support `enum`. - quote removal from `#include` directives as some compilers error out on any quote char present in non-comment code. This preprocess is limited to shared header files to avoid the temptation to use C++ syntax in .glsl files. See b42adab3a2 for more details. **Typenames** Using C++ typenames makes more sense in a C++ header. The GLSL translation layer define aliases to the GLSL types (`float4 > vec4`, `float4x4 > mat4`). This was chosen because having things like `using mat4 = float4x4` would make C++ code more complex and ambiguous to what type should be use in non shared files. The GLSL typenames are also quite ambiguous (ex: `vec3`, `mat4`, `dvec2`). **Math API** GLSL not being a Object Oriented language, we cannot rely on OOP function calls like `my_vector.normalize()` or `my_vector.normalized()`. Parsing and rewriting this syntax to functional syntax isn't trivial. For this reason, we are trying to stick to the functional syntax paradigm. We might not want stick to GLSL naming convention for consistency reason (BLI `invert()` vs. GLSL `inverse()`). These cases are easily handled by some `#define` on GLSL side. It is important to note that while the goal it to have a shared API, we might introduce some functionality only on the C++ side. Likewise, we can introduce some functions only on GLSL side if it makes sense. For vectors: [D13791: BLI: Refactor vector types & functions to use templates](https://archive.blender.org/developer/D13791) For matrix: [D16625: BLI: Refactor matrix types & functions to use templates](https://archive.blender.org/developer/D16625) **Padding** Vector types are still unaligned. Padding in struct definitions used in UBOs is still manual until we add similar alignment rules as GLSL. - `float3` / `(u)int3` still need manual padding and alignment to 16bytes. - `float3x3` is prohibited in shared struct declarations as they "alias" to a `float4x3` in GLSL. This might need be improved in the future. **C++ subset** The most commonly missed feature for now are: - Parameter reference `&` (which would translate to `inout` in GLSL). - No arrays (syntax differs greatly in GLSL). - No pointers or references. - No iterators. - No namespace. - No template. These are for shared GLSL/C++ code only. They are still allowed in the implementation of the API in C++ as long as the function call syntax can be emulated in GLSL. This might need be improved in the future. **Adoption** This project was motivated by the EEVEE-Next project (#93220) and the whole viewport module is moving in a similar direction (Workbench, DrawManager, Overlays...). Other project such as the new sculpt paint engine ([D16083](https://archive.blender.org/developer/D16083)) or geometry nodes have expressed their interest into moving computations to the GPU and this will help reduce friction. **Relevance:** It was brought up during the Metal Backend effort (#96261) that we might want to transition to HLSL for more friendly/modern shading language. This is currently out of scope as we still need to support OpenGL for a few more years until the vulkan backend takes over. After this, we might consider switching to HLSL and many of the issues highlighted above might be solved. **References:** https://docs.gl/sl4/acos https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-intrinsic-functions https://registry.khronos.org/OpenGL/specs/gl/GLSLangSpec.4.60.pdf

Clément Foucault commented

2022-12-08 14:31:44 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'