Include Python executable with Blender. #43486

New Issue

Campbell Barton · 2015-01-30T11:44:37+01:00

Campbell Barton commented

2015-01-30 11:44:37 +01:00

This tasks proposed to include python in Blender (python.exe on windows, so every Blender installation can be ensured to run the same version of Python that's included with Blender).

Motivation

Short Version: subprocesses.

Blender currently uses Python for many tasks, but anything that takes longer then a few seconds will lock Blender up, Currently some addons load a separate Blender process (SketchFab for example), However if this is only to transfer data, the overhead of running an entire Blender instance, just for its Python interpreter isn't very efficient.

Uses for Python subprocesses:

General online integration (file upload/download ... asset manager, cloud... etc).
Background processing (avoid locking up Blender).

Blender could just run Python, but this means you have to install Python and have it available in the systems PATH which isn't so common on Windows.

Including the python binary means we can ensure that there will be a python3.4 for certain.

Size considerations

On my system (64bit Linux) (python+libpython3.4) compress to ~1mb. It will vary between platforms but should stay under 2-3mb.

Platform maintainers

Since we already include a full python for each platform. Including the python binary in fact isn't much extra work (will be included in svn lib/(platform)/, will have to be copied across from the system).
Note that this will have to be a static build (since we aren't LD_PRELOADing libpython)

Access from Blender

Currently blender includes: 2.73/python/lib/python3.4

The python binary can be located at: 2.73/python/bin/python

We can have an attribute to access this to avoid each script having to find the path.
eg bpy.app.python_path. (as we have for bpy.app.binary_path to access Blender's binary)

For Linux distributions which don't bundle Python, this can simply point to $(which python)

This tasks proposed to include `python` in Blender (`python.exe` on windows, so every Blender installation can be ensured to run the same version of Python that's included with Blender). Motivation ---- *Short Version:* subprocesses. Blender currently uses Python for many tasks, but anything that takes longer then a few seconds will lock Blender up, Currently some addons load a separate Blender process (SketchFab for example), However if this is only to transfer data, the overhead of running an entire Blender instance, just for its Python interpreter isn't very efficient. Uses for Python subprocesses: - General online integration (file upload/download ... asset manager, cloud... etc). - Background processing (avoid locking up Blender). Blender could just run Python, but this means you have to install Python and have it available in the systems `PATH` which isn't so common on Windows. Including the `python` binary means we can ensure that there will be a `python3.4` for certain. Size considerations ---- On my system (64bit Linux) (`python`+`libpython3.4`) compress to ~1mb. It will vary between platforms but should stay under 2-3mb. Platform maintainers ---- Since we already include a full python for each platform. Including the python binary in fact isn't much extra work (will be included in svn `lib/(platform)/`, will have to be copied across from the system). Note that this will have to be a static build (since we aren't `LD_PRELOAD`ing `libpython`) Access from Blender ---- Currently blender includes: `2.73/python/lib/python3.4` The python binary can be located at: `2.73/python/bin/python` We can have an attribute to access this to avoid each script having to find the path. eg `bpy.app.python_path`. (as we have for `bpy.app.binary_path` to access Blender's binary) For Linux distributions which don't bundle Python, this can simply point to `$(which python)`

Campbell Barton commented

2015-01-30 11:44:37 +01:00

Changed status to: 'Open'

Campbell Barton commented

2015-01-30 11:44:37 +01:00

Added subscriber: @ideasman42

Bastien Montagne commented

2015-01-30 12:33:17 +01:00

Added subscriber: @mont29

Bastien Montagne commented

2015-01-30 12:33:17 +01:00

Wouldn't mind that, but I’m not a maintainer so… ;)

Anyway, my main question is: is it really worth it? Do we have use cases where it would really benefit to use lighter py process instead of full blender?

Here, empty py process is about 3MB, and empty blender process (in background) is about 37MB. This is much much bigger, but is it really an issue still? I would not expect more than a few of those processes running at once (usually no more than one actually), and since blender is already in mem, loading it again is really fast too. So unless you intend to start hundreds of processes in a very short time…

Wouldn't mind that, but I’m not a maintainer so… ;) Anyway, my main question is: is it really worth it? Do we have use cases where it would really benefit to use lighter py process instead of full blender? Here, empty py process is about 3MB, and empty blender process (in background) is about 37MB. This is much much bigger, but is it really an issue still? I would not expect more than a few of those processes running at once (usually no more than one actually), and since blender is already in mem, loading it again is really fast too. So unless you intend to start hundreds of processes in a very short time…

Dalai Felinto commented

2015-01-30 13:03:30 +01:00

Added subscriber: @dfelinto

Dalai Felinto commented

2015-01-30 13:03:30 +01:00

+1
As a user I can see the benefit of this as well. Do you think the Blenderplayer should have this too? Or should it be optional?

+1 As a user I can see the benefit of this as well. Do you think the Blenderplayer should have this too? Or should it be optional?

Campbell Barton commented

2015-01-30 13:16:48 +01:00

@mont29, yep, we can keep using blender just for py access, its not horrible but not really nice either, Since we are considering having asset managers that can download data in the background, I would rather this not be depending on an entire blender process.

@dfelinto, blenderplayer using same python as Blender, so some way for it to find the Python binary is trivial.

@mont29, yep, we can keep using blender just for py access, its not horrible but not really nice either, Since we are considering having asset managers that can download data in the background, I would rather this not be depending on an entire blender process. @dfelinto, blenderplayer using same python as Blender, so some way for it to find the Python binary is trivial.

Diego Gangl commented

2015-01-30 13:35:59 +01:00

Added subscriber: @Januz

Diego Gangl commented

2015-01-30 13:35:59 +01:00

+10000
I've recently implemented some code that works in the background for an addon, and had to use the Blender binary just for Python. A simple py script would be lighter, faster and it'd allow me to decouple it completely from Blender so users can call it from the cli (or other scripts).

You mention it has to be a static build. Is bpy.app.python_path always going to point to a python bin, or do we have to try it (and have a fallback)?

+10000 I've recently implemented some code that works in the background for an addon, and had to use the Blender binary just for Python. A simple py script would be lighter, faster and it'd allow me to decouple it completely from Blender so users can call it from the cli (or other scripts). You mention it has to be a static build. Is `bpy.app.python_path` always going to point to a python bin, or do we have to `try` it (and have a fallback)?

Lapineige commented

2015-01-30 16:12:02 +01:00

Added subscriber: @Lapineige

Campbell Barton commented

2015-01-30 17:02:05 +01:00

@Januz bpy.app.python_path Would always point to a valid python binary, weather on portable blender.org setup's or on linux distro's which include python.
Of course we cant help if someone manually deletes python from blender, but this isn't some case that needs to be supported - if someone manually removes files in Blender, it can be considered a broken installation.

A Python script writer should be able to call python in a single line of code, without searching for it.

@Januz `bpy.app.python_path` Would always point to a valid `python` binary, weather on portable blender.org setup's or on linux distro's which include python. Of course we cant help if someone manually deletes python from blender, but this isn't some case that needs to be supported - *if someone manually removes files in Blender, it can be considered a broken installation.* A Python script writer should be able to call python in a single line of code, without searching for it.

Jesse Sutherland commented

2015-02-01 06:25:06 +01:00

Added subscriber: @SynaGl0w

JT Nelson commented

2015-02-01 16:52:57 +01:00

Added subscriber: @jta

Greg Zaal commented

2015-02-01 17:25:44 +01:00

Added subscriber: @GregZaal

Dealga McArdle commented

2015-02-01 17:30:16 +01:00

Added subscriber: @tetha.z

Sebastian Brachi commented

2015-02-01 18:25:27 +01:00

Added subscriber: @Brachi

Inês Almeida commented

2015-02-03 13:22:39 +01:00

Added subscriber: @brita

Inês Almeida commented

2015-02-03 13:22:39 +01:00

@ideasman42 you want to have linux pointing to the system installed python? why? this could lead to mismatching python versions.

Campbell Barton commented

2015-02-03 22:11:28 +01:00

@brita re: system python.

I'm referring to packaged versions of Blender (as you would install on debian/redhad/fedora.... etc), these dont bundle Python.
We can assume python3.4 is in the PATH in these cases (We can use include defines to construct the path dynamically so Py3.5+ works too, whichever version of Python Blender is compiled against).

@brita re: system python. I'm referring to packaged versions of Blender (as you would install on debian/redhad/fedora.... etc), these dont bundle Python. We can assume `python3.4` is in the PATH in these cases (We can use include defines to construct the path dynamically so Py3.5+ works too, whichever version of Python Blender is compiled against).

Marcin Nijaki commented

2015-02-04 13:18:30 +01:00

Added subscriber: @marcino15

Marcin Nijaki commented

2015-02-04 13:18:30 +01:00

It will support pip?

Campbell Barton commented

2015-02-04 13:38:54 +01:00

@marcino15

Initially no, it wouldnt support pip, at the moment we remove distutils from the bundled Python installation.

Including it isn't especially difficult, it was just removed because there was no reason to have it and to save a bit of space.

Im not against this, but if we include distutils it may open us up to some maintenance issues. (users upgrading Blender and wanting to keep existing pip modules for eg).
So I would first like to evaluate the use-case for having pip with Blender to see if its really worth bothering with.

@marcino15 Initially no, it wouldnt support pip, at the moment we remove `distutils` from the bundled Python installation. Including it isn't especially difficult, it was just removed because there was no reason to have it and to save a bit of space. Im not against this, but if we include `distutils` it may open us up to some maintenance issues. (users upgrading Blender and wanting to keep existing pip modules for eg). So I would first like to evaluate the use-case for having pip with Blender to see if its really worth bothering with.

jens verwiebe commented

2015-02-11 08:24:27 +01:00

Added subscriber: @jensverwiebe

jens verwiebe commented

2015-02-11 08:24:27 +01:00

Thats easy doing and i see it's benefits.
I will provide a bin to our libs and adapt scons to place it in VERSION/python/bin

So we can test this and decide pro/contra much better in the end.

Jens

Thats easy doing and i see it's benefits. I will provide a bin to our libs and adapt scons to place it in VERSION/python/bin So we can test this and decide pro/contra much better in the end. Jens

Dalai Felinto commented

2015-03-03 15:10:19 +01:00

Is there any patch to test this? I believe this would fix issues like this: #43869

Germano Cavalcante commented

2015-04-17 22:10:46 +02:00

Added subscriber: @mano-wii

Sergey Sharybin commented

2015-04-19 16:33:28 +02:00

Added subscriber: @Sergey

Campbell Barton commented

2015-05-08 23:25:55 +02:00

@dfelinto, I dont think this makes any difference for #43869

Campbell Barton commented

2015-05-08 23:45:33 +02:00

Update. added bc2f77e1da

Now we only have to copy the executable over on install, and include in lib/

Update. added bc2f77e1da Now we only have to copy the executable over on install, and include in `lib/`

Dalai Felinto commented

2015-05-09 08:35:28 +02:00

@ideasman42 but if we are distributing the Python binary with Blender we can(/should?) also link Blender against it (which would take care of the forementioned issue).

That is basically what we do in OSX with -DWITH_PYTHON_FRAMEWORK.

@ideasman42 but if we are distributing the Python binary with Blender we can(/should?) also link Blender against it (which would take care of the forementioned issue). That is basically what we do in OSX with **-DWITH_PYTHON_FRAMEWORK**.

jens verwiebe commented

2015-05-09 18:46:05 +02:00

@ Dalai: the python binary i added to darwin libs is fully static against python and has nothing todo with python_framework.
Place it in the bundle in an apropiate way ( ....2.47/python/bin ) and you are done.
Now you must just call it instead of new blender py instances for subprocesses ....

Jens

@ Dalai: the python binary i added to darwin libs is fully static against python and has nothing todo with python_framework. Place it in the bundle in an apropiate way ( ....2.47/python/bin ) and you are done. Now you must just call it instead of new blender py instances for subprocesses .... Jens

Campbell Barton commented

2015-06-03 12:17:00 +02:00

Changed status from 'Open' to: 'Resolved'

Campbell Barton closed this issue

2015-06-03 12:17:00 +02:00

Campbell Barton self-assigned this 2015-06-03 12:17:00 +02:00

Campbell Barton commented

2015-06-03 12:17:00 +02:00

Added for linux/osx/windows closing.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Include Python executable with Blender. #43486

Motivation

Size considerations

Platform maintainers

Access from Blender