VFX Reference Platform 2021 Compatibility #83246
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
21 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#83246
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Since 2.92 will be released in 2021, it seems logical to upgrade to the libraries of the reference platform for that year.
https://vfxplatform.com/
However some of the libraries have not been released yet, and bcon1 ends in a few weeks. So I suggest we wait to upgrade for 2.93.
Here's the relevant libraries and what seems to me the required upgrades to match the platform:
Changed status from 'Needs Triage' to: 'Confirmed'
Added subscriber: @brecht
Added subscriber: @EAW
I expect OpenColorIO 2 to be the most complicated upgrade, I can help with that as the render module owner if needed.
OpenEXR 3 also has some significant changes, as Imath was separated into its own library. Changes here are probably more on the build system side than code.
Both these libraries only have an RC release, not a final one. Also OpenVDB 8 is not out yet. So we could also decide to postpone this upgrade to 2.93, since the LTS is the most important version to be compatible with the VFX reference platform.
It would be good to hear from platform maintainers about what they think is best to do here.
Added subscribers: @Jeroen-Bakker, @LazyDodo
Updating the deps builder for OCIO is easy, I have a patch somewhere already. given rather significant changes to the API the integration work if more than I'm willing to chew off, check with @Jeroen-Bakker he started looking at this last week.
Added subscriber: @mont29
So they stick to a 2.5 years old python again? Seriously...
I can't find any clear commitments about release dates, so I updated the description to suggest waiting for 2.93. OpenColorIO 2 is a good improvement for users, everything else probably does not really matter. Not enough reason to rush this I think.
It's indeed unfortunate that it's still on Python 3.7. A lot of software didn't fully transition to Python 3 yet even though it was in the 2020 platform, so they wanted to make it easier to get there in 2021.
@mont29 IIRC the VFX platform’s plan is sticking to 3.7 for 2021 and then jumping to 3.9 for 2022. I will post the reference when I find it.
All MSVC build libs are actually build with 2017 (v15.9) as newer compiler will seamlessly work with older libs, but that is a one way street, 2019 will link 2017 libs just fine, 2017 will not link 2019 libs, the same applies to subversions (ie 16.7 will link 16.4 libs, but the opposite may or may not work) so for maximum compatibility they are build with an older compiler. If this ever becomes an issue we can re-asses this, but for now we're fine.
Here is OIIO's PR to support building against OpenEXR 3.0 + Imath 3.0, in case looking at it is helpful to implement the build side changes needed for Blender.
Added subscriber: @AditiaA.Pratama
For OCIO there is an tmp-ocio-v2 branch on git.blender.org it is still work in progress (eg does not compile). The steps would first be to do a minimum migration (make it work), and by doing that find out what could be refactored as OCIO has now similar mechanisms. I did find the provided documentation still limited but have the idea what needs to be done.
+1 for releasing with the next lts release.
Added subscribers: @Ton, @dr.sybren
On the topic if Python versions: @Ton said on bf-committers the decision was to stick with the VFX Reference Platform for 2020 (also in #68774). Since we're planning for 2021 now: was there ever an official decision to stick with older Pythons for more than 2020? If so, I'd like to see that motivated and officially announced first. Otherwise I'd propose to go back to our original approach of simply following Python releases and move to version 3.9.
PS: note that Python 3.7 isn't even receiving bugfixes any more (source).
PPS: Ton has opened the discussion on bf-committers. I'll continue & build Python 3.7.9, as that's a good idea anyway regardless of whether we'll move to 3.newer later or not.
The task description (and also the VFX Platform site itself) mentions upgrading to OpenEXR 3.0.x, but on both the OpenEXR homepage and their GitHub the latest release seems to be 2.5.x. Even their master branch lists 2.5.99 as the in-development version. I don't see any release candidate mentioned in these places.
We are currently using version 2.4.0, and I think it's fine to upgrade, but I don't see how we can target 3.0.x for Blender 2.92.
Yes, OpenVDB 8.x and OpenColorIO 2.x are also not released yet. That's why I postponed this to 2.93.
Added subscriber: @ideasman42
Is now a good time to re-evaluate if we should stick with Python 3.7x? - as defined by the VFX platform.
Recently a user ran into a bug in Python 3.7x which has been fixed in later releases (#84752), while we could back-port fixes, I'd rather only do this as a last resort, especially since we can't rely on people to use our bundled Python when a system Python may be used.
If there is still interest to use Python 3.7x, we could keep using 3.7x for the 2.9x series. If we don't have any sign that there is a significant benefit and the VFX platform is still on a >1 year old Python, we could consider upgrading to the current Python release (now with annually major point releases).
Added subscriber: @Simon_I
Added subscriber: @skittlituted_scorpion
Ton asked about this on bf-committers and Twitter, and we got feedback from one studio saying that Python version compatibility was helpful, and no other feedback that I saw.
I think it's a matter of the Blender admins making a decision here. Personally I would stick to 3.7, but upgrading seems reasonable too.
There was not much positive feedback indeed; I am afraid we only get the negative feedback when we drop the VFX platform alignment. My preference would be to keep the alignment for now.
Did they give any insights into why this is the case:
Agree with @ideasman42 's questions.
Otherwise I see really little reasons to stick to a two years-old, unmaintained version of python just because VFX platform does not want to update it, when they are using bleeding edge versions for many of the other libraries. Even the requested linux compiler (gcc 9) is from March 2020!
This is the tweet: [[ https://twitter.com/jeff_hanna/status/1334507235985199109 | Yes. Aligning with Python releases has allowed Volition to build cross-DCC apps without having to worry about binary incompatibility on C-compiled Python libraries. Please consider continuing.
]]
I don't think the issue is that it's impossible to make things compatible, just that it's a pain to deal with multiple Python versions.
We should update Python, for sure, if only because of what @mont29 said: the 3.7 branch doesn't see bugfixes any more (source), it only receives security fixes.
The preference to stick to 3.7 sounds like a chicken-and-egg problem, where everybody wants to stick to the same Python version again until it's a year after its EOL. For Python 3.7 this is 2023-06-27, so if things keep going as they did when moving away from Python 2.7 my prediction is that the VFX Platform will stick to 3.7 until 2024.
Added subscriber: @rboxman
There was clear feedback/preference for 3.8 in 2021 already but the continued lag relating to updating from 2.7 complicated it. 3.9 was even mentioned as a potential target for 2022.
Read the thread here: https://groups.google.com/g/vfx-platform-discuss/c/-UfNwawZges
At some point we should do a cost/benefit analysis of the VFX platform requiring old versions of Python.
Regarding being stuck on older versions of Python. I don't think it matters exactly how old the version is - if it's 1-2+ years old Python which isn't getting bug-fixes then it's old enough for it to be considered a significant down-side. Especially if Blender users are running into problems which are resolved in newer Python version.
Unless we consider the check-box of supporting the VFX platform to be justification on it's own - at some point we should be able to see benefits to users.
That is, users getting tangible benefits beyond avoiding hypothetical breakages (for example - a studio integrating Blender into a pipeline that uses Python modules which are part of the VFX platform).
So far I think it's fair to say we didn't see this yet, maybe it's not been long enough.
While I'm not pushing to re-evaluate this decision just now, if there are no visible gains to speak of even after some years - the cons probably outweigh the pros.
I'd say just rip that band-aid off now, the vfxplatform appears to have a pattern of not being able to get their act together in the python department. They painted them selves into a corner with 2.7 and they are seemingly doing it again now with 3.7, But there is always the ever dangling carrot of "next year we'll have our act in order" followed by ".........probably......no guarantees..... we'll see... sooo..... uhmm... yeahh..... seemed expensive, so we decided to not do it...."
if we decide "The carrot is real, next year will be better!" fair enough won't stand in the way, Personally it's starting to feel like the carrot may just be a vfx effect, it should be ignored and only past decisions should be judged, which do not paint a very hopeful picture.
It's not like we're pushing for bleeding edge python here, we just want to update to a version that ...[checks notes].... "still fixes bugs" ? oof....
My two cents on this subject as I have come across this issue several times as well.
When importing a binary module in python on windows there is a linkage error when not compiling to the exact python you are targeting. On linux and macos I haven't noticed any issues here. What is a bit strange as normal Windows DLL system is better managed. This might be something that should be discussed with python. Of course since we stick to a single python version (2.80+) this hasn't been an issue anymore.
There are also some ways to work around this. eg compile the python wrapper separately from the actual binaries you want to import. Compiling wrappers should not be an issue for studios to guarantee that their logic is the same. The only variable is the wrapper and that is really easy to test (does it import). What won't work might also be incompatible with our licensing model. (eg loading third party DLLs directly in blender via python).
I would vote for 3.9 as the power of Blender is to be flexible. And open source allows forks with a different python version. This is somewhat following the VFX platform with an exception on python. I would argue that studios that benefit are able to maintain such a fork. I am not aware if the VFX ref platform officially allowing exceptions like other reference platforms do. Whatever we decide, we should bring this up with them.
OK let's upgrade and we tell the VFX platform to not be so ridiculously backward with acceptable current libraries.
Added subscriber: @dfelinto
As per their website, the 2022 plans will be presented around April :
Jan - Apr: Working Group discuss versions for next calendar year.We could stick to 3.7 until the 2.93 LTS (May). And then we can decide on whether to go to Python 3.9 (for Blender 3.x) knowing the plans by the VFX platform.
Not convinced shipping an LTS release with bugs we know can be fixed by just not following this somewhat strange standard - [x] is the way to go, as for the "lets just wait, the draft should be out soon" that is just the ever dangling carrot again, last year they announced they were "leaning towards Python 3.7 for CY2021 despite the clear support on this thread for 3.8." at the end of august.
By the way, I wrote my comment while Ton was writing his I believe. So I withdrawn my suggestion, I'm happy with his decision on it.
In this case I think we should move back tracking the current Python release.
That means we should not have to spend much time on discussions and justifications when keeping up with recent Python releases.
While there can be some exceptions (we might hold off if the new release is too close to Blender's release date, or if there are known problems), in general I think this served us fairly well in the past.
Added subscriber: @Sergey
What's our thoughts on updating to first corrective release (as an opposite of jumping straight into new series of major release)?
Does it make sense to stick to other libraries covered by the reference platform? Some of them are linked statically, others seems to have proper namespace separation.
I think we should look at this on a per-library basis. Alembic, for example, just upgrades to the latest available release, so that's all fine.
It's still helpful to match the platform as closely as possible, it's not just about Python. Why wouldn't we upgrade?
I'm not sure why static linking or namespace separation is relevant to this task.
Someone on chat pointed out python 3.9 does drop win7 support (retail date 2009, end of mainstream support ended in 2015, extended support ended jan 2020) I don't think this be a problem, but it is ultimately up to the BF to decide what they want to do here.
My personal stance is holding the python version back to follow a standard that made little sense was not in our best interest, the same applies to holding it back to support a 12 year old unsupported OS
Added subscriber: @erminos
Was suggesting the opposite: update as much as we need/can, without running into situation when the platform is pulling us down.
Compatibility with the pipeline. If someone uses a binary addon in one way or another, we don't run into symbol conflicts, meaning, we don't need to follow the platform letter by letter.
But this is indeed more of policy discussion,m rather than something for a task to track actual upgrade process. So lets move it somewhere else.
@Sergey, ah I see. The other libraries in the VFX reference platform are at the latest version or quite close to it, seems simpler to just follow it unless we have a specific problem to solve.
@brecht, Ok then. So newer Python, rest of the libraries we stay as close as we can to the platform.
@LazyDodo, Think it is not unfair to drop support of Windows 7. Is not receiving any updates, so is very hard to ensure quality working software there. Just once Python is updated please make sure the Requirements section of the download page is updated. Maybe mail the list as well (nowish?).
BTW, what about other libraries not listed by the VFX platform (like LLVM, OSL, etc.)?
Current OSL (1.10) is not building with latest LLVM 11 for example (Blender uses LLVM 9 currently).
OSL is being upgraded in D10212: Cmake/deps: Update OSL to 1.11.10.0. Haru is being added in D10280: CMake/Linux: Add libharu to platform_linux.cmake.
That's all the planned library changes I'm aware of for 2.93.
Added subscriber: @SteffenD
For future reference in case we want to re-evaluate our stance on following the VFX platform, the TBB and MSVC versions recommended in the VFX 2021 Platform are incompatible and requires patching of TBB to resolve some build issues. It's not a big issue for us since we ship all deps in svn, but still it's a very strange recommendation to make.
This issue was referenced by None@62551
Regarding OpenEXR 3.0, it now seems to be planned to be released on March 1.
https://lists.aswf.io/g/openexr-dev/message/4789
https://lists.aswf.io/g/openexr-dev/message/4790
That's not very far away. However we also expect that Alembic and OpenVDB libraries need build system update. OSL and OIIO master already have changes for OpenEXR 3.0, but we might still need to bump those versions as well. So if we have to wait for those releases, or patch things ourselves, it's getting complicated.
In practice, we are linking OpenEXR statically, symbols are hidden, and there are no file format changes that we need to upgrade for. For end-users and add-on developers, it should make no real difference which version we use internally.
So, I propose we stick to OpenEXR 2.x for Blender 2.93 LTS. If it turns out that a few weeks from now it's somehow easy to upgrade or we discover a good reason to do it, then we could reconsider. But I think 2.x is a good working assumption.
as much as i don't like messing with openexr since there's always issues, judging from the change logs i do suggest updating to v2.5.5
Updating to the latest 2.x is fine with me.
just to be sure we're not duplicating work, i'm on it.
edit: done! painless this time! D10416
That isn't the full change log.
This should be the full change log:
https://github.com/AcademySoftwareFoundation/openexr/blob/RB-2.5/CHANGES.md
But, *unfortunately,*it isn't.
This is the last commit:
4212416433Notice the changes to
IlmThread.h, IlmThread.cpp, and IlmThreadPool.cppThey are from:
https://github.com/AcademySoftwareFoundation/openexr/pull/921
Which is the result of Larry Gritz and Dieter Debaets trying to fix OIIO's CI on Windows.
https://github.com/OpenImageIO/oiio/issues/2797
Now I am not saying that the changes are bad, just that you might want to be aware of them/look them over given Blender's reliance on IlmThread.
Added subscriber: @ncannon
Added subscriber: @Frieder
Added subscriber: @Raimund58
Changed status from 'Confirmed' to: 'Resolved'
Added subscriber: @nanoSpawn
I hope this does not sound as beating on a dead horse. But wanted to point out something.
Renderman for Blender got just released. Renderman follows closely the VFX reference platform, and thus adheres to the current versions and specifications, that means Python 3.7, because it's the current specification.
Talking with people that works or has worked in the VFX industry, following the versioning matters because Python is not something each software bundles, and there's a close relationship between parts of the pipeline.
Renderman, for now, will only work on Blender 2.92 because the move to Python 3.9 broke VFX compatibility, they're committed to redo the bindings so Renderman woks on 2.93 on a future release of the renderer. But that's a hacky solution that requires tons of manual work. Totally not ideal.
Yes. I understand that a third party is no concern of the Blender Foundation. But the VFX Reference platform was created so it was easy to add software to a pipeline seamlessly and without needing ugly hacks or tons of manual work, and without compatibility problems.
So I launch a question. Why bother following a reference if at the end of the day no one bothers following it? As it stands now, no studio would use Blender on a VFX pipeline because it has the power of potentially breaking the compatibility.
I understand people wants to use the shiny last version of the libraries and that a 1-year old library feels like it was made in the 90s. But IIRC, Blender was attempting to move in an industry ready direction, becoming a solution that can be adopted by anyone for professional work, and the VFX reference platform is a huge step towards that.
Hi @nanoSpawn those decisions don't come lightly. You can read about the specifics about the Python version here: https://lists.blender.org/pipermail/bf-committers/2021-May/050990.html - 2.93 being a LTS and Python 3.7 no longer receiving updates were the determining factors in this case.
The renderman team is welcome to join our discussions and provide feedback. So far there has been very little specific feedback regarding the VFX platform adoption in our official channels (i.e., bf-committers). For instance, replies to this email: https://lists.blender.org/pipermail/bf-committers/2020-December/050799.html