Python can't handle German umlaut characters in strings #23197

New Issue

Christoph Peters · 2010-08-05T22:19:48+02:00

Christoph Peters commented

2010-08-05 22:19:48 +02:00

%%%The Blender UI lets you enter the German Umlaut characters ü, Ü, ö, Ö, ä, Ä and ß. However strings containing them cause exceptions whenever Python scripts try to work with them. This is probably a more general problem, but I got it with Umlauts.

Steps to reproduction:

Download and open the attached BlenderUmlautBug.blend.
Hit Run Script for the script that is opened (ReproduceBugScript).
You should get the following output in the Blender console window:
Traceback (most recent call last):

File "I:\Downloads\BlenderUmlautBug.blend/ReproduceBugScript", line 2, in <module>

UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 0: unexpected end of data

location::-1

The cause is that the script tries to print the name of the cube mesh and this name is "ü".

System:

CPU: Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz (2 CPUs), ~1.9GHz
RAM: 2048 MB
GFX: ATI Radeon HD 2600 XT
OS: German Version of Windows 7 Professional 64-bit (6.1, Build 7600) (7600.win7_gdr.100226-1909)
Blender: 2.53.0 r'30581'
Python: Python 3.1.2 (r312:79149, Mar 20 2010, 22:55:39) [MSC v.1500 64 bit (AMD64)] on win32
%%%

%%%The Blender UI lets you enter the German Umlaut characters ü, Ü, ö, Ö, ä, Ä and ß. However strings containing them cause exceptions whenever Python scripts try to work with them. This is probably a more general problem, but I got it with Umlauts. Steps to reproduction: 1) Download and open the attached BlenderUmlautBug.blend. 2) Hit Run Script for the script that is opened (ReproduceBugScript). You should get the following output in the Blender console window: Traceback (most recent call last): ``` File "I:\Downloads\BlenderUmlautBug.blend/ReproduceBugScript", line 2, in <module> ``` UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 0: unexpected end of data location:<unknown location>:-1 location:<unknown location>:-1 The cause is that the script tries to print the name of the cube mesh and this name is "ü". System: * CPU: Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz (2 CPUs), ~1.9GHz * RAM: 2048 MB * GFX: ATI Radeon HD 2600 XT * OS: German Version of Windows 7 Professional 64-bit (6.1, Build 7600) (7600.win7_gdr.100226-1909) * Blender: 2.53.0 r'30581' * Python: Python 3.1.2 (r312:79149, Mar 20 2010, 22:55:39) [MSC v.1500 64 bit (AMD64)] on win32 %%%

Christoph Peters commented

2010-08-05 22:19:48 +02:00

Changed status to: 'Open'

Christoph Peters commented

2010-08-05 22:22:44 +02:00

%%%Sorry, I placed this in the wrong tracker. Of course it belongs into the Blender 2.5 Bug Tracker.%%%

Brecht Van Lommel commented

2010-08-22 12:16:13 +02:00

%%%Closed duplicate report:
http://projects.blender.org/tracker/index.php?func=detail&aid=23424&group_id=9&atid=498%%%

%%%Closed duplicate report: http://projects.blender.org/tracker/index.php?func=detail&aid=23424&group_id=9&atid=498%%%

Campbell Barton commented

2010-08-30 10:24:23 +02:00

%%%fixed r31655.%%%

Campbell Barton commented

2010-08-30 10:24:24 +02:00

Changed status from 'Open' to: 'Resolved'

Campbell Barton closed this issue

2010-08-30 10:24:24 +02:00

Jeducious commented

2017-04-29 03:01:44 +02:00

Added subscriber: @JamesCrowther

Jeducious commented

2017-04-29 03:01:44 +02:00

Changed status from 'Resolved' to: 'Open'

Jeducious reopened this issue

2017-04-29 03:01:44 +02:00

Jeducious commented

2017-04-29 03:01:44 +02:00

Hi guys, using current release (see system info here-> system-info.txt) blender 2.78c on Macosx 10.12 and believe this issue has returned as far as the title is concerned, attempting to access materials from the python console that have names containing umlauts causes errors, for example

Using this blend file from blendswap -> http://www.blendswap.com/blends/view/2259

Trying to access the materials from the python console window in blender like so...

>>>D.materials['Seitengänge_unten]


Traceback (most recent call last):
  File "/Applications/Blender/blender.app/Contents/Resources/2.78/python/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<blender_console>", line 1, in <module>
KeyError: 'bpy_prop_collection[key]: key "Seitengänge_unten" not found'

Also, when recursively printing all the materials in the console...

>>>for m in D.materials:

...     print(m)
...     
<bpy_struct, Material("Augen1")>
<bpy_struct, Material("Augen1.001")>
<bpy_struct, Material("Boden")>
<bpy_struct, Material("Boden_kacheln")>
<bpy_struct, Material("Fenster")>
<bpy_struct, Material("Fenster_klein")>
<bpy_struct, Material("Fensterrahmen")>
<bpy_struct, Material("Feuerschalen")>
<bpy_struct, Material("Halo")>
<bpy_struct, Material("Kohle")>
<bpy_struct, Material("Material")>
<bpy_struct, Material("Material.001")>
<bpy_struct, Material("Material.003")>
<bpy_struct, Material("Material.005")>
<bpy_struct, Material("Material.006")>
<bpy_struct, Material("Schale")>
<bpy_struct, Material("Seiteng�nge_unten")>
<bpy_struct, Material("Steine")>
<bpy_struct, Material("S�ulen")>
<bpy_struct, Material("S�uleN_DUNKEL")>
<bpy_struct, Material("S�ulen_vorne")>
<bpy_struct, Material("Treppe")>

and lastly, i am running a the following code block in the python console which uses the python builtin 'hasattr', when it encounters materials whose names contain umlauts an error is thrown ...

>>>for m in D.materials:

...     if hasattr(m, 'name'):
...         print(m)
...         


<bpy_struct, Material("Augen1")>
<bpy_struct, Material("Augen1.001")>
<bpy_struct, Material("Boden")>
<bpy_struct, Material("Boden_kacheln")>
<bpy_struct, Material("Fenster")>
<bpy_struct, Material("Fenster_klein")>
<bpy_struct, Material("Fensterrahmen")>
<bpy_struct, Material("Feuerschalen")>
<bpy_struct, Material("Halo")>
<bpy_struct, Material("Kohle")>
<bpy_struct, Material("Material")>
<bpy_struct, Material("Material.001")>
<bpy_struct, Material("Material.003")>
<bpy_struct, Material("Material.005")>
<bpy_struct, Material("Material.006")>
<bpy_struct, Material("Schale")>

Traceback (most recent call last):
  File "/Applications/Blender/blender.app/Contents/Resources/2.78/python/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<blender_console>", line 2, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 7: invalid continuation byte

Hi guys, using current release (see system info here-> [system-info.txt](https://archive.blender.org/developer/F578392/system-info.txt)) blender 2.78c on Macosx 10.12 and believe this issue has returned as far as the title is concerned, attempting to access materials from the python console that have names containing umlauts causes errors, for example Using this blend file from blendswap -> http://www.blendswap.com/blends/view/2259 Trying to access the materials from the python console window in blender like so... ``` >>>D.materials['Seitengänge_unten] Traceback (most recent call last): File "/Applications/Blender/blender.app/Contents/Resources/2.78/python/lib/python3.5/code.py", line 91, in runcode exec(code, self.locals) File "<blender_console>", line 1, in <module> KeyError: 'bpy_prop_collection[key]: key "Seitengänge_unten" not found' ``` Also, when recursively printing all the materials in the console... ``` >>>for m in D.materials: ... print(m) ... <bpy_struct, Material("Augen1")> <bpy_struct, Material("Augen1.001")> <bpy_struct, Material("Boden")> <bpy_struct, Material("Boden_kacheln")> <bpy_struct, Material("Fenster")> <bpy_struct, Material("Fenster_klein")> <bpy_struct, Material("Fensterrahmen")> <bpy_struct, Material("Feuerschalen")> <bpy_struct, Material("Halo")> <bpy_struct, Material("Kohle")> <bpy_struct, Material("Material")> <bpy_struct, Material("Material.001")> <bpy_struct, Material("Material.003")> <bpy_struct, Material("Material.005")> <bpy_struct, Material("Material.006")> <bpy_struct, Material("Schale")> <bpy_struct, Material("Seiteng�nge_unten")> <bpy_struct, Material("Steine")> <bpy_struct, Material("S�ulen")> <bpy_struct, Material("S�uleN_DUNKEL")> <bpy_struct, Material("S�ulen_vorne")> <bpy_struct, Material("Treppe")> ``` and lastly, i am running a the following code block in the python console which uses the python builtin 'hasattr', when it encounters materials whose names contain umlauts an error is thrown ... ``` >>>for m in D.materials: ... if hasattr(m, 'name'): ... print(m) ... <bpy_struct, Material("Augen1")> <bpy_struct, Material("Augen1.001")> <bpy_struct, Material("Boden")> <bpy_struct, Material("Boden_kacheln")> <bpy_struct, Material("Fenster")> <bpy_struct, Material("Fenster_klein")> <bpy_struct, Material("Fensterrahmen")> <bpy_struct, Material("Feuerschalen")> <bpy_struct, Material("Halo")> <bpy_struct, Material("Kohle")> <bpy_struct, Material("Material")> <bpy_struct, Material("Material.001")> <bpy_struct, Material("Material.003")> <bpy_struct, Material("Material.005")> <bpy_struct, Material("Material.006")> <bpy_struct, Material("Schale")> Traceback (most recent call last): File "/Applications/Blender/blender.app/Contents/Resources/2.78/python/lib/python3.5/code.py", line 91, in runcode exec(code, self.locals) File "<blender_console>", line 2, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 7: invalid continuation byte ```

Campbell Barton commented

2017-05-01 08:05:47 +02:00

@JamesCrowther, this is a different issue to the original report.

The problem is that existing files may have non-UTF8 names which we don't support anymore.

To solve this we would need to rename id-names on loading files, but thats quite error prone too.
(would need to check new-name is unique, might break library linking).

@JamesCrowther, this is a different issue to the original report. The problem is that existing files may have non-UTF8 names which we don't support anymore. To solve this we would need to rename id-names on loading files, but thats quite error prone too. *(would need to check new-name is unique, might break library linking).*

Jeducious commented

2017-05-01 08:23:21 +02:00

@ideasman42 , different issue - no problem. Just seemed close enough to warrant re-opening this one rather than creating a whole new bug report.

The problem is that existing files may have non-UTF8 names which we don't support anymore.

Will this affect the ability to open old files in future versions? Its been a long standing feature of blender to be able to open old files with practically any version of blender and (mostly) the file opens ok.

Is there a way to avoid this error in python somehow? In many cases, I've used hasattr to make sure an attribute I am going to access exists so I don't cause an error. I suppose one way to avoid it might be to revert to try/except?

But, in that case, what remedial action can a script take? Changing the name is one approach I guess we could take, but as you said, this might cause errors downstream.

Any ideas?

@ideasman42 , different issue - no problem. Just seemed close enough to warrant re-opening this one rather than creating a whole new bug report. > The problem is that existing files may have non-UTF8 names which we don't support anymore. Will this affect the ability to open old files in future versions? Its been a long standing feature of blender to be able to open old files with practically any version of blender and (mostly) the file opens ok. Is there a way to avoid this error in python somehow? In many cases, I've used hasattr to make sure an attribute I am going to access exists so I don't cause an error. I suppose one way to avoid it might be to revert to try/except? But, in that case, what remedial action can a script take? Changing the name is one approach I guess we could take, but as you said, this might cause errors downstream. Any ideas?

Bastien Montagne commented

2017-05-03 12:11:40 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2017-05-03 12:11:40 +02:00

Changed status from 'Open' to: 'Resolved'

Bastien Montagne closed this issue

2017-05-03 12:11:40 +02:00

Bastien Montagne commented

2017-05-03 12:11:40 +02:00

Yes, Blender tries to keep backward compatibility as much as possible - but we cannot address all (corner)cases, even less when we are talking about 8 years old files…

Also here file is not actually broken, it’s access from py API which becomes problematic. You most likely can work something around, Python's encoding module is quite powerful and flexible, you can also control how encoding errors are handled, etc.

Thanks for the report, but this is more like support request for an old broken corner case, not a bug anyway.

Yes, Blender tries to keep backward compatibility as much as possible - but we cannot address all (corner)cases, even less when we are talking about 8 years old files… Also here file is not actually broken, it’s access from py API which becomes problematic. You most likely can work something around, [Python's encoding module](https://docs.python.org/3.5/library/codecs.html#encodings-and-unicode) is quite powerful and flexible, you can also control how encoding errors are handled, etc. Thanks for the report, but this is more like support request for an old broken corner case, not a bug anyway.

Campbell Barton commented

2017-05-03 12:24:32 +02:00

Posted solution to update your files: https://blender.stackexchange.com/questions/78931

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Python can't handle German umlaut characters in strings #23197