Fix for Python failure with .blend files loaded from paths including non-ASCII characters #35176

New Issue

Tamito Kajiyama · 2013-05-02T02:43:03+02:00

Tamito Kajiyama commented

2013-05-02 02:43:03 +02:00

%%%Test platform: 64-bit Windows Vista, with the active code page set to 932 (Japanese Shift_JIS)
Test Blender revision: 56458

Problem: Freestyle fails when .blend files are loaded from paths including non-ASCII characters (e.g., Japanese).
File loading is okay (thanks to the trunk revision 56454), but Freestyle rendering results in a Python error shown below:

UnicodeDecodeError: 'mbcs' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page.

The error comes from Py_CompileString() called from python_script_exec() in source/blender/python/intern/bpy_interface.c.
The second argument of Py_CompileString() must be a byte array in the default file system encoding (Py_FileSystemDefaultEncoding).
In the test platform, Py_FileSystemDefaultEncoding is "mbcs" (i.e., Windows-specific class of encodings).

Apparently, G.main->name is a byte array in the UTF-8 encoding (please confirm this).
This variable indicates the fully qualified name of the .blend file being opened and manipulated by Blender.
The .blend file name is then embedded in a string (fn_dummy) to be passed as the 2nd argument of Py_CompileString().

In summary, the UnicodeDecodeError above is resulting from an attempt in Py_CompileString() to decode a UTF-8 byte array as if
the byte array were in the MBCS encoding.

Attached patch is intended to fix the reported issue. Code review and testing are much appreciated.%%%

%%%Test platform: 64-bit Windows Vista, with the active code page set to 932 (Japanese Shift_JIS) Test Blender revision: 56458 Problem: Freestyle fails when .blend files are loaded from paths including non-ASCII characters (e.g., Japanese). File loading is okay (thanks to the trunk revision 56454), but Freestyle rendering results in a Python error shown below: UnicodeDecodeError: 'mbcs' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page. The error comes from Py_CompileString() called from python_script_exec() in source/blender/python/intern/bpy_interface.c. The second argument of Py_CompileString() must be a byte array in the default file system encoding (Py_FileSystemDefaultEncoding). In the test platform, Py_FileSystemDefaultEncoding is "mbcs" (i.e., Windows-specific class of encodings). Apparently, G.main->name is a byte array in the UTF-8 encoding (please confirm this). This variable indicates the fully qualified name of the .blend file being opened and manipulated by Blender. The .blend file name is then embedded in a string (fn_dummy) to be passed as the 2nd argument of Py_CompileString(). In summary, the UnicodeDecodeError above is resulting from an attempt in Py_CompileString() to decode a UTF-8 byte array as if the byte array were in the MBCS encoding. Attached patch is intended to fix the reported issue. Code review and testing are much appreciated.%%%

Tamito Kajiyama commented

2013-05-02 02:43:03 +02:00

Changed status to: 'Open'

Brecht Van Lommel commented

2013-05-02 03:56:24 +02:00

%%%Campbell is the expert on this, so assigning to him.

G.main->name is indeed UTF-8, and the python documentation mentions Py_CompileString expects a string in the file system encoding, so this change makes sense to me. PyRun_File in the same function looks like it needs the same conversion to the file system encoding?

Probably it would be good to split this into a utility function that converts a char* from UTF-8 to filesystem default encoding so it can be reused in different places.
%%%

%%%Campbell is the expert on this, so assigning to him. G.main->name is indeed UTF-8, and the python documentation mentions Py_CompileString expects a string in the file system encoding, so this change makes sense to me. PyRun_File in the same function looks like it needs the same conversion to the file system encoding? Probably it would be good to split this into a utility function that converts a char* from UTF-8 to filesystem default encoding so it can be reused in different places. %%%

Tamito Kajiyama commented

2013-05-03 19:47:41 +02:00

%%%Hi Brecht and Campbell,

Thank you for the confirmation and positive response on this matter. Following the code review comments from Brecht, I have duly updated the patch set and made it more comprehensive. There are actually three places in the code base where file names in the file system default encoding are necessary (instead of UTF-8 as it is so far). It is noted that the identified string encoding inconsistency affects the following ways of Python script execution.

Execution of Freestyle scripts.
Execution of Python scripts using the command line option -P.
Execution of text datablocks (marked as Python modules) at start-up.
Reloading of imported modules using imp.reload().

The attached patch accounts for all these execution methods, allowing: (a) .blend files and Python scripts to reside in paths including non-ASCII characters; and (b) text datablocks to have non-ASCII ID names.

Since the trunk revision 56454 is likely included in the upcoming 2.67 release and file loading from non-ASCII paths will be okay, the reported issue will affect many Blender applications including Freestyle rendering. A timely fix of the issue would hence be greatly appreciated.%%%

%%%Hi Brecht and Campbell, Thank you for the confirmation and positive response on this matter. Following the code review comments from Brecht, I have duly updated the patch set and made it more comprehensive. There are actually three places in the code base where file names in the file system default encoding are necessary (instead of UTF-8 as it is so far). It is noted that the identified string encoding inconsistency affects the following ways of Python script execution. - Execution of Freestyle scripts. - Execution of Python scripts using the command line option -P. - Execution of text datablocks (marked as Python modules) at start-up. - Reloading of imported modules using imp.reload(). The attached patch accounts for all these execution methods, allowing: (a) .blend files and Python scripts to reside in paths including non-ASCII characters; and (b) text datablocks to have non-ASCII ID names. Since the trunk revision 56454 is likely included in the upcoming 2.67 release and file loading from non-ASCII paths will be okay, the reported issue will affect many Blender applications including Freestyle rendering. A timely fix of the issue would hence be greatly appreciated.%%%

Brecht Van Lommel commented

2013-05-04 00:01:02 +02:00

%%%About the PyC_EncodeFSDefault implementation. I found there exists a PyUnicode_EncodeFSDefault in the python API, and is already used in Blender. The implementation is here: http://hg.python.org/cpython/file/56ca8eb5207a/Objects/unicodeobject.c

Perhaps we should just always call that instead to be sure it does the right thing, since your version seems to be a bit different? I'm not sure of how much using e.g. "surrogateescape" instead of "strict" matters in practice, but maybe it matters in some corners cases.

ALso the strcpy should be replaced by BLI_strncpy.%%%

%%%About the PyC_EncodeFSDefault implementation. I found there exists a PyUnicode_EncodeFSDefault in the python API, and is already used in Blender. The implementation is here: http://hg.python.org/cpython/file/56ca8eb5207a/Objects/unicodeobject.c Perhaps we should just always call that instead to be sure it does the right thing, since your version seems to be a bit different? I'm not sure of how much using e.g. "surrogateescape" instead of "strict" matters in practice, but maybe it matters in some corners cases. ALso the strcpy should be replaced by BLI_strncpy.%%%

Tamito Kajiyama commented

2013-05-04 09:22:14 +02:00

%%%Hi Brecht and Campbell,

Thanks for the comments. I agree that PyUnicode_EncodeFSDefault would be better. Also BLI_strncpy is used instead of strcpy. I hope the updated patch set is okay now.%%%

%%%Hi Brecht and Campbell, Thanks for the comments. I agree that PyUnicode_EncodeFSDefault would be better. Also BLI_strncpy is used instead of strcpy. I hope the updated patch set is okay now.%%%

Shinsuke Irie commented

2014-01-27 06:43:34 +01:00

Added subscriber: @IRIEShinsuke

Shinsuke Irie commented

2014-01-27 06:43:34 +01:00

Any update on this? Many users want the problem to be resolved.

Tamito Kajiyama commented

2014-01-28 16:34:57 +01:00

I keep seeing many Japanese Blender users suffering from this issue.

For ease of further tests and code review I have just added D267. Any action on the patch set is much appreciated.

I keep seeing many Japanese Blender users suffering from this issue. For ease of further tests and code review I have just added [D267](https://archive.blender.org/developer/D267). Any action on the patch set is much appreciated.

Campbell Barton commented

2014-01-30 08:17:26 +01:00

Checking on this, note that filepaths in blender aren't strictly utf-8, for rna filepaths which are not utf8 we need to support these already.

Tamito Kajiyama commented

2014-01-30 09:41:11 +01:00

I am afraid I missed the point. What do you mean by being not strictly utf-8?

The only requirement for the patch is that G.main->name and the ID name of text data blocks be encoded in UTF-8.
Are there cases where they are not encoded in UTF-8? If so, how do we know the encoding?

I am afraid I missed the point. What do you mean by being not strictly utf-8? The only requirement for the patch is that G.main->name and the ID name of text data blocks be encoded in UTF-8. Are there cases where they are not encoded in UTF-8? If so, how do we know the encoding?

Tamito Kajiyama commented

2014-01-31 17:10:51 +01:00

I had a conversation with Campbell on IRC. Here is a brief summary of the discussion.

The main concern is that the PyC_EncodeFSDefault() function I wrote may return NULL when the given string is not valid as a UTF-8 string, so that the conversion to the file system default encoding fails. Since callers of the function may not expect NULL, it would be nicer to return some string instead of NULL. It is recalled that the input string to PyC_EncodeFSDefault() is bpy.data.filepath + os.sep + text.name and what may cause the failure of encoding conversion is the first part. Two options were discussed:

PyC_UnicodeFromByte() may provide a solution.
Just omit the troublesome bpy.data.filepath and return text.name as a fallback.

Another concern is that some paths cannot be opened with byte strings at all (cf. BLI_fopen() and intern/utfconv/utf_winfunc.c). This may happen and there is no obvious solution for it.

I had a conversation with Campbell on IRC. Here is a brief summary of the discussion. The main concern is that the PyC_EncodeFSDefault() function I wrote may return NULL when the given string is not valid as a UTF-8 string, so that the conversion to the file system default encoding fails. Since callers of the function may not expect NULL, it would be nicer to return some string instead of NULL. It is recalled that the input string to PyC_EncodeFSDefault() is bpy.data.filepath + os.sep + text.name and what may cause the failure of encoding conversion is the first part. Two options were discussed: 1. PyC_UnicodeFromByte() may provide a solution. 2. Just omit the troublesome bpy.data.filepath and return text.name as a fallback. Another concern is that some paths cannot be opened with byte strings at all (cf. BLI_fopen() and intern/utfconv/utf_winfunc.c). This may happen and there is no obvious solution for it.

Tamito Kajiyama commented

2014-01-31 17:23:28 +01:00

Correction:

Another concern is that some paths cannot be opened with byte strings at all (cf. BLI_fopen() and intern/utfconv/utf_winfunc.c).

I meant on Windows.

Correction: > Another concern is that some paths cannot be opened with byte strings at all (cf. BLI_fopen() and intern/utfconv/utf_winfunc.c). I meant on Windows.

Martijn Berger commented

2014-02-09 16:28:21 +01:00

Added subscriber: @MartijnBerger

Sv. Lockal commented

2014-02-11 07:46:57 +01:00

Added subscriber: @Lockal

Tamito Kajiyama commented

2014-02-13 14:22:50 +01:00

I have updated D267 based on the discussions with Campbell concerning a junk UTF-8 string as input. Now the helper function PyC_EncodeFSDefault() (added by the patch) always return a valid string rather than NULL, so that callers of the function won't fail even when they don't expect NULL.

The chosen solution here is just to return the input string as it is when encoding conversion cannot be done.

I have examined several other fall back solutions, including:

PyUnicode_DecodeLatin1()
PyUnicode_DecodeRawUnicodeEscape()
PyUnicode_DecodeUnicodeEscape()
for conversion from bytes to unicode, and
PyUnicode_AsLatin1String()
PyUnicode_AsRawUnicodeEscapeString()
PyUnicode_AsUnicodeEscapeString()
for conversion from unicode to bytes. The Latin-1 decoder and encoder pass through all bytes, so they are equivalent to a simple copy of the input string to output. The UnicodeEscape and RawUnicodeEscape functions involve the additional handling of backslash escaped characters such as \xNN and \uNNNN. Since backslash is a directory separator on Windows, using these functions may further complicate error conditions.

Since a junk UTF-8 string cannot be a valid path anyway, it looks like the only possibility is to give up encoding conversion and fall back to BLI_strncpy().

I have updated [D267](https://archive.blender.org/developer/D267) based on the discussions with Campbell concerning a junk UTF-8 string as input. Now the helper function PyC_EncodeFSDefault() (added by the patch) always return a valid string rather than NULL, so that callers of the function won't fail even when they don't expect NULL. The chosen solution here is just to return the input string as it is when encoding conversion cannot be done. I have examined several other fall back solutions, including: - PyUnicode_DecodeLatin1() - PyUnicode_DecodeRawUnicodeEscape() - PyUnicode_DecodeUnicodeEscape() for conversion from bytes to unicode, and - PyUnicode_AsLatin1String() - PyUnicode_AsRawUnicodeEscapeString() - PyUnicode_AsUnicodeEscapeString() for conversion from unicode to bytes. The Latin-1 decoder and encoder pass through all bytes, so they are equivalent to a simple copy of the input string to output. The UnicodeEscape and RawUnicodeEscape functions involve the additional handling of backslash escaped characters such as \xNN and \uNNNN. Since backslash is a directory separator on Windows, using these functions may further complicate error conditions. Since a junk UTF-8 string cannot be a valid path anyway, it looks like the only possibility is to give up encoding conversion and fall back to BLI_strncpy().

Tamito Kajiyama changed title from ~~Fix for Freestyle failure with .blend files loaded from paths including non-ASCII characters~~ to Fix for Python failure with .blend files loaded from paths including non-ASCII characters

2014-02-13 14:23:42 +01:00

Tamito Kajiyama commented

2014-02-13 14:24:56 +01:00

Just changed the title, since the problem being addressed here is not Freestyle specific but related to Python in general.

Campbell Barton commented

2014-02-14 12:29:28 +01:00

I've been looking into this bug and oddly enough I can't redo it (my fs encoding is mbcs), but I've tried to use text which isnt mbcs compatible and its not givine a python error.

I'd still rather avoid adding new string conversion functions - since we can get bpy.data.filepathalready.

Note, we already ran into similar problems here.
http://bugs.python.org/issue9713

Committed change to merge text compile into a single function.

Attached a patch which I think fixes the problem for the text editor, by using Py_CompileStringObject so we can convert the string into a PyObject first (using the same method used to get bpy.data.filepath).

@kjym3 could you check if this works for you?

unicode_text.diff

Note, this doesn't deal with bpy_interface.c. - just compiling text.

I've been looking into this bug and oddly enough I can't redo it (my fs encoding is `mbcs`), but I've tried to use text which isnt mbcs compatible and its not givine a python error. I'd still rather avoid adding new string conversion functions - since we can get `bpy.data.filepath`already. Note, we already ran into similar problems here. http://bugs.python.org/issue9713 Committed change to merge text compile into a single function. Attached a patch which I think fixes the problem for the text editor, by using `Py_CompileStringObject` so we can convert the string into a PyObject first (using the same method used to get `bpy.data.filepath`). @kjym3 could you check if this works for you? [unicode_text.diff](https://archive.blender.org/developer/F77245/unicode_text.diff) Note, this doesn't deal with `bpy_interface.c`. - just compiling text.

Tamito Kajiyama commented

2014-02-16 13:59:22 +01:00

@ideasman42
The changes by unicode_text.diff look okay. Now that we rely on Py_CompileStringObject() that takes a file name in the form of a Unicode object, we don't have to deal with the file system default encoding.

For Freestyle, we have to make similar changes to python_script_exec() called from BPY_text_exec().

It is noted that Py_CompileStringObject() is new in Python 3.4. Windows binaries of this version are not in the lib svn repository. I just built Python 3.4c1 myself from the tarball using VS 2008 to test the patch.

Below find a revised version of your patch set, including your original changes plus mine to get Freestyle working with non-ASCII file paths.

unicode_text_v2.diff

@ideasman42 The changes by unicode_text.diff look okay. Now that we rely on Py_CompileStringObject() that takes a file name in the form of a Unicode object, we don't have to deal with the file system default encoding. For Freestyle, we have to make similar changes to python_script_exec() called from BPY_text_exec(). It is noted that Py_CompileStringObject() is new in Python 3.4. Windows binaries of this version are not in the lib svn repository. I just built Python 3.4c1 myself from the tarball using VS 2008 to test the patch. Below find a revised version of your patch set, including your original changes plus mine to get Freestyle working with non-ASCII file paths. [unicode_text_v2.diff](https://archive.blender.org/developer/F77515/unicode_text_v2.diff)

Campbell Barton commented

2014-02-17 02:00:19 +01:00

So OK to postpone this fix until 2.71? (or whenever we bundle Py3.4).

Tamito Kajiyama commented

2014-02-17 05:08:21 +01:00

Yes (personally I prefer to have this fix asap though).

Thomas Dinges commented

2014-02-21 16:18:06 +01:00

Added subscriber: @ThomasDinges

Thomas Dinges commented

2014-02-21 16:18:06 +01:00

As we won't include Python 3.4 for Blender 2.70, removing Blender 2.70 here.

blender-admin commented

2014-04-30 15:43:01 +02:00

This issue was referenced by blender/blender-addons-contrib@4d1a109dde

This issue was referenced by blender/blender-addons-contrib@4d1a109ddec02ad7e527d8b65a5cdc8d4a7528a9

blender-admin commented

2014-04-30 15:43:01 +02:00

This issue was referenced by 4d1a109dde

This issue was referenced by 4d1a109ddec02ad7e527d8b65a5cdc8d4a7528a9

Campbell Barton commented

2014-04-30 15:47:39 +02:00

Changed status from 'Open' to: 'Resolved'

Campbell Barton closed this issue

2014-04-30 15:47:39 +02:00

Campbell Barton commented

2014-04-30 15:47:39 +02:00

Closed by commit 4d1a109dde.

Closed by commit 4d1a109dde.

Tamito Kajiyama commented

2014-06-13 04:30:25 +02:00

Just realized that the problem has been partly left not addressed. Please, consider reviewing the patch just added to the task.

Tamito Kajiyama commented

2014-06-13 04:44:39 +02:00

Changed status from 'Resolved' to: 'Open'

Tamito Kajiyama reopened this issue

2014-06-13 04:44:39 +02:00

Tamito Kajiyama commented

2014-06-13 04:49:10 +02:00

py-exec-fix-test.blend

Here is a quick .blend file for testing. Just press the "Run Script" button in the text editor. Without the proposed fix you will see a stack trace printed in the console (after the completion of the script execution).

[py-exec-fix-test.blend](https://archive.blender.org/developer/F94103/py-exec-fix-test.blend) Here is a quick .blend file for testing. Just press the "Run Script" button in the text editor. Without the proposed fix you will see a stack trace printed in the console (after the completion of the script execution).

Campbell Barton commented

2014-06-16 07:15:04 +02:00

@kjym3. I can't redo the bug (MSVC2013, Windows7)

I tried to run and to register and both work fine. - print "hello" with no exception.

EDIT, somehow Missed D595, looks good.

@kjym3. I can't redo the bug (MSVC2013, Windows7) I tried to run and to register and both work fine. - print "hello" with no exception. ---- EDIT, somehow Missed [D595](https://archive.blender.org/developer/D595), looks good.

Campbell Barton commented

2014-06-16 07:30:38 +02:00

Changed status from 'Open' to: 'Resolved'

Campbell Barton closed this issue

2014-06-16 07:30:38 +02:00

Campbell Barton commented

2014-06-16 07:30:38 +02:00

Committed, resolved.

Tamito Kajiyama commented

2014-06-16 07:43:31 +02:00

Great, thanks!

Tony Mullen commented

2014-06-23 06:16:30 +02:00

Added subscriber: @TonyMullen

Tony Mullen commented

2014-06-23 06:16:30 +02:00

I am experiencing a related problem in the latest buildbot build as of Jun 19, (b49e6d0) (and all previous builds as far back as 2.65 that I've tried) on Japanese Windows 8 on a Vaio Ultrabook. Python fails on startup and I am unable to find any workaround to run Blender with its menus, etc intact (I've tried putting the Blender install directly under C:\ so as to avoid Japanese characters in the path, but no luck).

This also gives me the same error as described above in the console:

UnicodeDecodeError: 'mbcs' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page.

Should I open a new report for this?

Tony

Fwiw, I'm adding the screenshot I took of the Traceback.

I am experiencing a related problem in the latest buildbot build as of Jun 19, (b49e6d0) (and all previous builds as far back as 2.65 that I've tried) on Japanese Windows 8 on a Vaio Ultrabook. Python fails on startup and I am unable to find any workaround to run Blender with its menus, etc intact (I've tried putting the Blender install directly under C:\ so as to avoid Japanese characters in the path, but no luck). This also gives me the same error as described above in the console: UnicodeDecodeError: 'mbcs' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page. Should I open a new report for this? Tony Fwiw, I'm adding the screenshot I took of the Traceback. ![blenderscreenshot.PNG](https://archive.blender.org/developer/F95316/blenderscreenshot.PNG)

Tamito Kajiyama commented

2014-06-23 07:55:32 +02:00

Exactly the same error occurs when the name of the user running Blender contains non-ASCII MBCS-compatible characters such as those in Japanese. Patch D604 is intended to address this issue. The reported problem is different from what #35176 was meant to address, so filing another bug report is appreciated.

Exactly the same error occurs when the name of the user running Blender contains non-ASCII MBCS-compatible characters such as those in Japanese. Patch [D604](https://archive.blender.org/developer/D604) is intended to address this issue. The reported problem is different from what #35176 was meant to address, so filing another bug report is appreciated.

Tony Mullen commented

2014-06-23 08:29:56 +02:00

Ok, thanks. Created a separate bug report.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Fix for Python failure with .blend files loaded from paths including non-ASCII characters #35176