+1 for this. Currently my GafferCycles implementation needs all elements to be persistent by default, and I currently get around this by managing the scene object vectors outside of ccl::scene and making sure to clear the pointers out of Cycles and transplant them back in eg.
https://github.com/boberfly/GafferCycles/blob/master/src/GafferCycles/IECoreCyclesPreview/Renderer.cpp#L2865

I had this posted a day or two ago but I didn't understand how phabricator's differential works. I've made a basic inline comment in bvh_embree.h in which the new function needs fixing to compile properly:

I'm taking a look at this patch to test with GafferCycles but just the Cycles part, but I am seeing if I can merge it with the ImageTextureNode. From what I understand currently, to fill up the tiles you are relying on Blender to fill this up? The approach I am testing is to just let the user pass a filename with the format 'file.<udim>.tx' and then deriving the name at the SVMCompile/OSLCompile stage. Does anyone have an opinion on this? Also should there be an additional projection called 'udim' or 'mari' which works like the standard 'flat' only it explicitly tells the code to treat it as a udim mapping?

Just an update, I managed to build the latest Blender 2.80 here (I had /usr/bin/c++ & cc set to clang not gcc, doh!)

Something which would be great would be a procedural API in the same vein as Renderman and Arnold, where one can dynamically load geometry to the renderer when the renderer hits a bounding box ray hit for the first time.