Page MenuHome

Using PyPI from Blender
Open, NormalPublic

Description

Proposal

Improve the workflow for installing Python modules from the Python package index PyPI.

This is a design task to evaluate options.

Note:

Motivation

For Python developers to be able to install 3rd party modules into Blender.

While this is possible, it's quite involved:
https://blender.stackexchange.com/questions/56011/how-to-install-pip-for-blenders-bundled-python

This task is to evaluate possible solutions:

  • Use pip with Blender's built-in Python.
  • Only support pip for system-wide Python installations.

Package Compatibility

  • Windows Python MSVC compatibility. (EDIT: this is no longer an issue).

    In the past, Blender's Python was built with a different MSVC version, making binary modules incompatible.
  • File System Permissions

    The Blender's Python directory may not be writable.

    This could be solved using the --user argument.

    eg: on Linux pip install my_package --user installs to ~/.local/lib/python3.8/site-packages/bintrees

    Possible Solution: Set PYTHONUSERBASE to a directory under Blender's own configuration.

    This has the advantage that it will work with both a system wide & a bundled Python installation.

    See: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUSERBASE

Details

Type
Design

Related Objects

Event Timeline

I support bundling pip with Blender. It would have helped me in quite a few cases. I did not have any issues with binary compatibility after installing it myself yet. That does not mean that there are none of course.

For my vscode extension for Blender development, I had to use pip to install some packages in Blender. It works in most cases, but I remember that it did not work in every case. I would have to dig deeper to figure out the failure cases more precisely.

The code responsible for installing the packages is here: https://github.com/JacquesLucke/blender_vscode/blob/master/pythonFiles/include/blender_vscode/installation.py

This solution does not require any user interaction. Blender only has to be started from within vscode and everything else should happen automatically.

I would also appreciate to have pip in blender. I'm struggling with using numpy or scipy inside of blender. It would make it so much easier to use scientific data for visualising in blender.

At Airbus Defence & Space, we are using ensurepip (and then pip install) to install 3rd party packages inside Blender, for addon and pipeline development, based on buildin python from blender. Hope we can continue to perform it even on PC without system install of python

@mudu: numpy is already bundled with blender (but scipy not, but can be install with pip as described in stackexchange link.

It would be great if the improved workflow would make use of venv. This would allow a nice replacement for how modules were handled through the Scripts path in the preferences with the Blender specific directory structure.

If we consider improving the workflow of how Python modules are installed in Blender, I'd like to make the following proposal:

Instead of installing modules directly into Blender's bundled Python, let the user create, reference and use venvs in Blender's preferences. Blender would create the venv based on the bundled Python and therefore modules installed through pip in that venv would be compatible to Blender's bundled Python. The user could select one of the venvs to use, however Blender doesn't actually load the venv since it already has it's own interpreter running. Instead it checks if the referenced venv interpreter matches the bundled Python version and if it's ok, loads the modules. There could also be a section in the preferences that allow to install modules through pip into the venv within Blender.

The benefits of this approach:

  • The bundled Python remains in a clean state
  • If you want to install modules that have conflicting dependencies you can simply create separate venvs and switch between them
  • If the Python version of a new version Blender remains the same, you can re-use the venv. In case it doesn't match you can detect this and report it to the user, instead of having issues where modules aren't working properly.
  • Different venvs could be created for different needs
  • Unlike when using the Scripts path you don't have to manually copy modules from a manually created venv into a modules folder
  • Provides a single consistent workflow for working with Python modules

This could possibly replace the Scripts path in the preferences altogether, since custom startup files can already be created through application templates and add-ons would be installed in the add-ons section of the preferences. I know it's more work that simply adding pip, however it may be a cleaner solution in the long run.

BTW a better link for this discussion would be: https://blender.stackexchange.com/questions/5287/using-3rd-party-python-modules.

With ensurepip it's actually not really complicated.

Package Compatibility
Windows Python MSVC compatibility.
In the past, Blender's Python was built with a different MSVC version, making binary modules incompatible.
Is this still an issue?

Good news: No longer an issue, pip works and you can use it to install things !

Bad news: I think by default pip wants to install into the scripts folder, however if blender is installed into c:\program files\ writing there maybe problematic and require administrator rights, which is guaranteed to be a troublesome in most corporate and educational environments.

@LazyDodo (LazyDodo) You're right pip is already shipped with 2.81. So this task is basically solved?

Bad news: I think by default pip wants to install into the scripts folder, however if blender is installed into c:\program files\ writing there maybe problematic and require administrator rights, which is guaranteed to be a troublesome in most corporate and educational environments.

It looks like we can work-around this problem using PYTHONUSERBASE and pip's --user argument to install packages in a path we define (noted in task).

@Robert Guetzkow (rjg), I'm not sure if we need a venv if we can define the user-site directory.

While I haven't used venv much, and never dug into it's internals - a user-site directory looks to have most of the advantages noted in your post about venv's.

This comment was removed by Robert Guetzkow (rjg).

@Campbell Barton (campbellbarton) If setting the PYTHONUSERBASE allows to switch between multiple directories in order to separate the modules per directory from each other and from the system-wide installed modules, then it would be a good solution. It would also have less overhead than a venv.