Blender complains that it cannot find "stdosl.h", despite it existing at "/usr/share/OSL/shaders/stdosl.h" #73830

New Issue

Kyle · 2020-02-15T10:58:19+01:00

Kyle commented

2020-02-15 10:58:19 +01:00

System Information
Operating system: Arch Linux
Graphics card: RX 580

Blender Version
Broken: df1e9b662b, "Cleanup: Fix build warnings from OSL shader compilation"
Worked: 6a28d14f72, "Fix #73740: Auto-key "Only Available" de-selects f-curves"

Short description of error
Blender cannot find "stdosl.h", even though it exists.

Exact steps for others to reproduce the error
On Arch Linux, install openshadinglanguage, and then attempt to compile Blender master.

Other

Related: https://developer.blender.org/D6812

Upon examining the borked commit, I noticed this comment:

FindOSL.cmake does not give us the location of the shader library but generally it can be figured out from the location of the oslc compiler. However if this fails you can set OSL_SHADER_DIR to bypass the auto discovery.

Unfortunately, there is no provided CMake option allowing a custom OSL_SHADER_DIR location to be set, as the current assumptions don't account for where packagers may place system files. In Arch Linux, the compiler is here, "/usr/bin/oslc", while the shader directory is here, "/usr/share/OSL/shaders".

**System Information** Operating system: Arch Linux Graphics card: RX 580 **Blender Version** Broken: df1e9b662bd6938f74579cea9d30341f3b6dd02b, "Cleanup: Fix build warnings from OSL shader compilation" Worked: 6a28d14f7236b3e6fbe7409da9ee916878d65feb, "Fix #73740: Auto-key "Only Available" de-selects f-curves" **Short description of error** Blender cannot find "stdosl.h", even though it exists. **Exact steps for others to reproduce the error** On Arch Linux, install openshadinglanguage, and then attempt to compile Blender master. **Other** Related: https://developer.blender.org/D6812 Upon examining the borked commit, I noticed this comment: > FindOSL.cmake does not give us the location of the shader library but generally it can be figured out from the location of the oslc compiler. However if this fails you can set OSL_SHADER_DIR to bypass the auto discovery. Unfortunately, there is no provided CMake option allowing a custom OSL_SHADER_DIR location to be set, as the current assumptions don't account for where packagers may place system files. In Arch Linux, the compiler is here, "/usr/bin/oslc", while the shader directory is here, "/usr/share/OSL/shaders".

Kyle commented

2020-02-15 10:58:20 +01:00

Added subscriber: @KyleDeVir

Brecht Van Lommel commented

2020-02-15 11:18:31 +01:00

Added subscribers: @LazyDodo, @brecht

Brecht Van Lommel commented

2020-02-15 11:18:31 +01:00

@LazyDodo, probably we should improve FindOpenShadingLanguage.cmake to find the shaders directory.

@LazyDodo, probably we should improve `FindOpenShadingLanguage.cmake` to find the shaders directory.

Ray molenkamp commented

2020-02-15 18:33:23 +01:00

I'm not on linux so i can't really test this but can anyone give this a whirl?

diff --git a/build_files/cmake/Modules/FindOpenShadingLanguage.cmake b/build_files/cmake/Modules/FindOpenShadingLanguage.cmake
index 01ed72051f7..72e40e74895 100644
--- a/build_files/cmake/Modules/FindOpenShadingLanguage.cmake
+++ b/build_files/cmake/Modules/FindOpenShadingLanguage.cmake
@@ -85,6 +85,17 @@ IF(OSL_FOUND)
          "\\1" OSL_LIBRARY_VERSION_MINOR ${OSL_LIBRARY_VERSION_MINOR})
 ENDIF(OSL_FOUND)

+FIND_PATH(OSL_SHADER_DIR
+  NAMES
+    stdosl.h
+  HINTS
+    ${OSL_ROOT_DIR}
+    ${ENV{OSLHOME}}
+    /usr/share/osl
+  PATH_SUFFIXES
+    shaders
+)
+
 MARK_AS_ADVANCED(
   OSL_INCLUDE_DIR
 )

I'm not on linux so i can't really test this but can anyone give this a whirl? ``` diff --git a/build_files/cmake/Modules/FindOpenShadingLanguage.cmake b/build_files/cmake/Modules/FindOpenShadingLanguage.cmake index 01ed72051f7..72e40e74895 100644 --- a/build_files/cmake/Modules/FindOpenShadingLanguage.cmake +++ b/build_files/cmake/Modules/FindOpenShadingLanguage.cmake @@ -85,6 +85,17 @@ IF(OSL_FOUND) "\\1" OSL_LIBRARY_VERSION_MINOR ${OSL_LIBRARY_VERSION_MINOR}) ENDIF(OSL_FOUND) +FIND_PATH(OSL_SHADER_DIR + NAMES + stdosl.h + HINTS + ${OSL_ROOT_DIR} + ${ENV{OSLHOME}} + /usr/share/osl + PATH_SUFFIXES + shaders +) + MARK_AS_ADVANCED( OSL_INCLUDE_DIR ) ````

Kyle commented

2020-02-16 01:44:53 +01:00

CMake doesn't like it:

CMake Error at build_files/cmake/Modules/FindOpenShadingLanguage.cmake:88 (FIND_PATH):

Syntax error in cmake code at

/tmp/makepkg/blender-git/src/blender/build_files/cmake/Modules/FindOpenShadingLanguage.cmake:93

when parsing string

${ENV{OSLHOME}}

Invalid character ('{') in a variable name: 'ENV'

Call Stack (most recent call first):

build_files/cmake/platform/platform_unix.cmake:71 (find_package)
build_files/cmake/platform/platform_unix.cmake:248 (find_package_wrapper)
CMakeLists.txt:801 (include)



  - Configuring incomplete, errors occurred!```

CMake doesn't like it: ```-- Found OSL: /usr/lib/liboslcomp.so;/usr/lib/liboslexec.so;/usr/lib/liboslquery.so CMake Error at build_files/cmake/Modules/FindOpenShadingLanguage.cmake:88 (FIND_PATH): ``` Syntax error in cmake code at ``` ``` /tmp/makepkg/blender-git/src/blender/build_files/cmake/Modules/FindOpenShadingLanguage.cmake:93 ``` ``` when parsing string ``` ``` ${ENV{OSLHOME}} ``` ``` Invalid character ('{') in a variable name: 'ENV' ``` Call Stack (most recent call first): ``` build_files/cmake/platform/platform_unix.cmake:71 (find_package) build_files/cmake/platform/platform_unix.cmake:248 (find_package_wrapper) CMakeLists.txt:801 (include) ``` - Configuring incomplete, errors occurred!```

Ray molenkamp commented

2020-02-16 03:09:32 +01:00

toss that one line for now then, just to see if it picks up your headers in /usr/share

Garry R. Osgood commented

2020-02-16 13:02:04 +01:00

Added subscriber: @grosgood

Garry R. Osgood commented

2020-02-16 13:02:05 +01:00

This bit me. Gentoo Linux here, with ##media-libs/osl-1.10.5## providing shaders in system folder ##/usr/include/OSL/shaders##
For what it is worth, my workaround:
- Define in build directory ##CMakeCache.txt## the symbol ## OSL_SHADER_DIR:PATH=/usr/include/OSL/shaders##
- Modified blender/intern/cycles/kernel/shaders/CMakeLists.txt:

  diff --git a/intern/cycles/kernel/shaders/CMakeLists.txt b/intern/cycles/kernel/shaders/CMakeLists.txt
  index 1c9445107ad..b1604978cce 100644
  --- a/intern/cycles/kernel/shaders/CMakeLists.txt
  +++ b/intern/cycles/kernel/shaders/CMakeLists.txt
  @@ -131,7 +131,7 @@ foreach(_file ${SRC_OSL})
     string(REPLACE ${CMAKE_SOURCE_DIR} ${CMAKE_BINARY_DIR} _OSO_FILE ${_OSO_FILE})
     add_custom_command(
       OUTPUT ${_OSO_FILE}
  -    COMMAND ${OSL_COMPILER} -q -O2  -I"${CMAKE_CURRENT_SOURCE_DIR}" -o ${_OSO_FILE} ${_OSL_FILE}
  +    COMMAND ${OSL_COMPILER} -q -O2  -I"${CMAKE_CURRENT_SOURCE_DIR}" -I"${OSL_SHADER_DIR}" -o ${_OSO_FILE} ${_OSL_FILE}
       DEPENDS ${_OSL_FILE} ${SRC_OSL_HEADERS} ${OSL_COMPILER})
     list(APPEND SRC_OSO
       ${_OSO_FILE}

TL;DR I added a second include directive so that the OSL compiler looks in the installed OSL shader directory as well as the specific blender build directory.

Haven't tested if this approach works beyond my peculiar little world. Hope it helps point the way to a more general solution.

[edit]
@KyleDeVir

Unfortunately, there is no provided CMake option...

Try: ##cmake ../blender -D OSL_SHADER_DIR:PATH=/usr/share/OSL/shaders##
Maybe substitute ##../blender## with the path to your blender source tree. In my case, the build directory is a co-directory of the source directory.

This bit me. Gentoo Linux here, with ##media-libs/osl-1.10.5## providing shaders in system folder ##/usr/include/OSL/shaders## For what it is worth, my workaround: - Define in build directory ##CMakeCache.txt## the symbol ## OSL_SHADER_DIR:PATH=/usr/include/OSL/shaders## - Modified [blender/intern/cycles/kernel/shaders/CMakeLists.txt](https://developer.blender.org/diffusion/B/browse/master/intern/cycles/kernel/shaders/CMakeLists.txt$134): ``` diff --git a/intern/cycles/kernel/shaders/CMakeLists.txt b/intern/cycles/kernel/shaders/CMakeLists.txt index 1c9445107ad..b1604978cce 100644 --- a/intern/cycles/kernel/shaders/CMakeLists.txt +++ b/intern/cycles/kernel/shaders/CMakeLists.txt @@ -131,7 +131,7 @@ foreach(_file ${SRC_OSL}) string(REPLACE ${CMAKE_SOURCE_DIR} ${CMAKE_BINARY_DIR} _OSO_FILE ${_OSO_FILE}) add_custom_command( OUTPUT ${_OSO_FILE} - COMMAND ${OSL_COMPILER} -q -O2 -I"${CMAKE_CURRENT_SOURCE_DIR}" -o ${_OSO_FILE} ${_OSL_FILE} + COMMAND ${OSL_COMPILER} -q -O2 -I"${CMAKE_CURRENT_SOURCE_DIR}" -I"${OSL_SHADER_DIR}" -o ${_OSO_FILE} ${_OSL_FILE} DEPENDS ${_OSL_FILE} ${SRC_OSL_HEADERS} ${OSL_COMPILER}) list(APPEND SRC_OSO ${_OSO_FILE} ``` **TL;DR** I added a second include directive so that the OSL compiler looks in the installed OSL shader directory as well as the specific blender build directory. Haven't tested if this approach works beyond my peculiar little world. Hope it helps point the way to a more general solution. [edit] @KyleDeVir > Unfortunately, there is no provided CMake option... Try: ##cmake ../blender -D OSL_SHADER_DIR:PATH=/usr/share/OSL/shaders## Maybe substitute ##../blender## with the path to your blender source tree. In my case, the build directory is a co-directory of the source directory.

Kyle commented

2020-02-16 13:32:17 +01:00

Your patch fails, sadly.

@LazyDodo It would probably be a good idea to allow packagers to set the OSL_SHADER_DIR variable themselves, as the comment in the OP states.

Yet, I think you may have simply forgotten to offer that as a CMake option, as there's currently zero way to set it.

Your patch fails, sadly. @LazyDodo It would probably be a good idea to allow packagers to set the `OSL_SHADER_DIR` variable themselves, as the comment in the OP states. Yet, I think you may have simply forgotten to offer that as a CMake option, as there's currently zero way to set it.

Kyle commented

2020-02-16 14:02:44 +01:00

@LazyDodo Ah, I see... your patch had /usr/share/osl, when it should be /usr/share/OSL, as most Linux filesystems are distinguish between lower and uppercase.

Actually, no, nevermind, that fails, too...

Hmmmmm......

@LazyDodo Ah, I see... your patch had `/usr/share/osl`, when it should be `/usr/share/OSL`, as most Linux filesystems are distinguish between lower and uppercase. Actually, no, nevermind, that fails, too... Hmmmmm......

Garry R. Osgood commented

2020-02-16 14:12:58 +01:00

@KyleDeVir
Note that my workaround is peculiar to a Gentoo linux box; the steps need a certain amount of remapping to work on other distros. Fortunately, Arch is not that far removed from Gentoo.
In your case, introduce a CMake symbol via

[blender_build_dir]$ cmake -S <path to your blender source tree> -B <path to your blender_build_dir> -D OSL_SHADER_DIR:PATH=<path to your installed OSL shaders, i.e. /usr/share/OSL/shaders>

Patch ##/intern/cycles/kernel/shaders/CMakeLists.txt## as indicated in my first comment
Probably be a good idea to create a working branch first to keep your git repository tidy

$ git branch --no-track osl_fix master
$ git checkout osl_fix

Then patch ##CMakeLists.txt##

This is just a Linux-centric workaround. Unfortunately, various Linux distros put the shader directory in various places. Witness Gentoo: ##/usr/include/OSL/shaders## vs. Arch: ##/usr/share/OSL/shaders##. It's an all-around PITA to deal with this. Probably, to duck this issue, is why blender developers from ages ago made their private copies, the issue that the good, brave @LazyDodo is trying to address.

@KyleDeVir Note that my workaround is peculiar to a Gentoo linux box; the steps need a certain amount of remapping to work on other distros. Fortunately, Arch is not that far removed from Gentoo. In your case, introduce a CMake symbol via ``` [blender_build_dir]$ cmake -S <path to your blender source tree> -B <path to your blender_build_dir> -D OSL_SHADER_DIR:PATH=<path to your installed OSL shaders, i.e. /usr/share/OSL/shaders> ``` Patch ##/intern/cycles/kernel/shaders/CMakeLists.txt## as indicated in my first comment Probably be a good idea to create a working branch first to keep your git repository tidy ``` $ git branch --no-track osl_fix master $ git checkout osl_fix ``` Then patch ##CMakeLists.txt## This is just a Linux-centric workaround. Unfortunately, various Linux distros put the shader directory in various places. Witness Gentoo: ##/usr/include/OSL/shaders## vs. Arch: ##/usr/share/OSL/shaders##. It's an all-around PITA to deal with this. Probably, to duck this issue, is why blender developers from ages ago made their private copies, the issue that the good, brave @LazyDodo is trying to address.

Kyle commented

2020-02-16 14:13:09 +01:00

Huh ~ it works, if I symlink /usr/share/OSL/shaders to /usr/shaders. That's... something. Not what's needed, anyhow. :|

Huh ~ it works, if I symlink `/usr/share/OSL/shaders` to `/usr/shaders`. That's... something. Not what's needed, anyhow. :|

Kyle commented

2020-02-16 14:16:42 +01:00

@grosgood

Thanks! Will give it a try.

@grosgood Thanks! Will give it a try.

Kyle commented

2020-02-16 14:30:14 +01:00

Yeah, your solution works. Cheers. :)

Garry R. Osgood commented

2020-02-16 20:19:14 +01:00

Great! Don't forget this is a hack - unofficial, unsupported, probably radioactive. If you've followed my suggestion of creating a local git branch ##osl_fix## to contain the patch, and you sit on the ##osl_fix## branch to do your builds, then you've got (just a little) extra work to keep ##osl_fix## aligned with ##master## whenever you perform ##make update##. See Creating Patches from Local Git Branches . The gist of it is: from ##osl_fix##: ##git checkout master##, ##make update##, ##git checkout osl_fix##, ##git rebase master##. When the fix for this lands in master, you can just checkout master and do the update, then abandon ##osl_fix:## ##git branch -D osl_fix##.

Great! Don't forget this is a hack - unofficial, unsupported, probably radioactive. If you've followed my suggestion of creating a local git branch ##osl_fix## to contain the patch, and you sit on the ##osl_fix## branch to do your builds, then you've got (just a little) extra work to keep ##osl_fix## aligned with ##master## whenever you perform ##make update##. See [Creating Patches from Local Git Branches ](https://wiki.blender.org/wiki/Tools/Patches#Creating_Patches_from_Local_Git_Branches). The gist of it is: from ##osl_fix##: ##git checkout master##, ##make update##, ##git checkout osl_fix##, ##git rebase master##. When the fix for this lands in master, you can just checkout master and do the update, then abandon ##osl_fix:## ##git branch -D osl_fix##.

Ray molenkamp commented

2020-02-16 22:08:59 +01:00

I manged to test some of this out on WSL , can you guys test the patch in D6865 ? i included both the arch and gentoo search paths so it should work out of the box now.

I manged to test some of this out on WSL , can you guys test the patch in [D6865](https://archive.blender.org/developer/D6865) ? i included both the arch and gentoo search paths so it should work out of the box now.

Garry R. Osgood commented

2020-02-16 22:22:11 +01:00

Yep - but not now. up to neck in alligators again. Never can tell what comes out of a Brooklyn sewer. Late this evening probably; post results in the wee hours Monday (-5 UTC, less wee +1 UTC).
Thanks thee, for the rapid turnaround!

Yep - but not now. up to neck in alligators again. Never can tell what comes out of a Brooklyn sewer. Late this evening probably; post results in the wee hours Monday (-5 UTC, less wee +1 UTC). Thanks thee, for the rapid turnaround!

Ray molenkamp commented

2020-02-16 22:53:45 +01:00

Wait, what?! The 'again' is troublesome since it seems to indicate that this is a common occurrence....

Garry R. Osgood commented

2020-02-16 23:38:41 +01:00

Different, unrelated alligators. These are pink. Alligators related to build systems are putrid green, have warts, big teeth. Pink ones have been dispatched now. Starting cleanup...

Garry R. Osgood commented

2020-02-17 00:33:25 +01:00

@LazyDodo
D6865 plays fine here. Completely uneventful. Ran a couple of builds, one Release and one Debug from empty build directories. Deleted ##CMakeCache.txt## to make sure no strange symbols were floating around. Running ctest now; nothing out of the ordinary there either, up to now.
Current to: version: 2.83 (sub 3), branch: osl_rayfix, commit date: 2020-02-16 22:53, hash: f1a13b1ab8d6, type: Release

@LazyDodo [D6865](https://archive.blender.org/developer/D6865) plays fine here. Completely uneventful. Ran a couple of builds, one Release and one Debug from empty build directories. Deleted ##CMakeCache.txt## to make sure no strange symbols were floating around. Running ctest now; nothing out of the ordinary there either, up to now. Current to: version: 2.83 (sub 3), branch: osl_rayfix, commit date: 2020-02-16 22:53, hash: f1a13b1ab8d6, type: Release

Kyle commented

2020-02-17 04:23:46 +01:00

Thanks! Will test. :)