Keymap is not exported with the correct encoding #82292

New Issue

EMM · 2020-11-01T04:08:49+01:00

EMM commented

2020-11-01 04:08:49 +01:00

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: GeForce RTX 2070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 452.06

Blender Version
Broken: version: 2.90.1, branch: master, commit date: 2020-09-23 06:43, hash: 3e85bb34d0
Worked: (newest version of Blender that worked as expected)

Short description of error
The exported keymap is not encoded with UTF-8 but an encoding depending on the locale. This produces a file which cannot be properly imported afterwards.

The keyconfig_export_as_data should specify the encoding when using open().

Exact steps for others to reproduce the error

Change your locale to e.g. Simplified Chinese.
Use the operator script to create an entry in the keymap.
Change the string property of the operator in the keymap to Chinese characters.
Export the keymap.

operator-to-reproduce-bug.py

**System Information** Operating system: Windows-10-10.0.19041-SP0 64 Bits Graphics card: GeForce RTX 2070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 452.06 **Blender Version** Broken: version: 2.90.1, branch: master, commit date: 2020-09-23 06:43, hash: `3e85bb34d0` Worked: (newest version of Blender that worked as expected) **Short description of error** The exported keymap is not encoded with UTF-8 but an encoding depending on the locale. This produces a file which cannot be properly imported afterwards. The `keyconfig_export_as_data` should specify the encoding when using `open()`. **Exact steps for others to reproduce the error** - Change your locale to e.g. Simplified Chinese. - Use the operator script to create an entry in the keymap. - Change the string property of the operator in the keymap to Chinese characters. - Export the keymap. [operator-to-reproduce-bug.py](https://archive.blender.org/developer/F9181126/operator-to-reproduce-bug.py)

EMM commented

2020-11-01 04:08:49 +01:00

Added subscriber: @Guai_Wo_Ge_EMM

Robert Guetzkow commented

2020-11-01 11:05:57 +01:00

Added subscriber: @rjg

Robert Guetzkow commented

2020-11-01 11:05:57 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Robert Guetzkow commented

2020-11-01 11:05:57 +01:00

The files you've attached appear to be encoded with GB2312 for simplified Chinese. Blender requires these files to be encoded with UTF-8. Additionally the keymap.py appears to be generally broken with NULL bytes at the beginning and seemingly corrupted output at the end of the file.

Could you please explain how you created these files and the exact steps to reproduce the issue? If Blender has saved these files that can't be imported, then this would be a bug. However, should you have manually edited the keymap and saved it with the unsupported encoding, then this is not a bug in Blender.

The files you've attached appear to be encoded with GB2312 for simplified Chinese. Blender requires these files to be encoded with UTF-8. Additionally the `keymap.py` appears to be generally broken with `NULL` bytes at the beginning and seemingly corrupted output at the end of the file. Could you please explain how you created these files and the exact steps to reproduce the issue? If Blender has saved these files that can't be imported, then this would be a bug. However, should you have manually edited the keymap and saved it with the unsupported encoding, then this is not a bug in Blender.

EMM commented

2020-11-01 12:23:07 +01:00

I didn't do other extra operations, I clicked on export, then immediately click on import, then it appears this error, and the two errors are different, I don't know what else to pay attention to, could it be because I use the Chinese system to cause it,Also attached is a video of me in action.
I've also found that creating a new keymap and selecting it also causes a bug!
EMM.py

QQ录屏20201101191923.mp4
我没有做其它多余的操作,我点击导出之后,立刻点击导入,然后它就出现了这样的报错,而且两个报错还不同,我也不知道还有其它什么地方注意的,会不会是因为我使用的中文系统导致的呢,另外附上我操作的视频
我还发现一个问题,就是新建一个 keymap 选择它也会出现BUG

- I didn't do other extra operations, I clicked on export, then immediately click on import, then it appears this error, and the two errors are different, I don't know what else to pay attention to, could it be because I use the Chinese system to cause it,Also attached is a video of me in action. - I've also found that creating a new keymap and selecting it also causes a bug! [EMM.py](https://archive.blender.org/developer/F9150289/EMM.py) [QQ录屏20201101191923.mp4](https://archive.blender.org/developer/F9150290/QQ录屏20201101191923.mp4) 我没有做其它多余的操作,我点击导出之后,立刻点击导入,然后它就出现了这样的报错,而且两个报错还不同,我也不知道还有其它什么地方注意的,会不会是因为我使用的中文系统导致的呢,另外附上我操作的视频我还发现一个问题,就是新建一个 keymap 选择它也会出现BUG

Robert Guetzkow commented

2020-11-01 13:47:19 +01:00

@Guai_Wo_Ge_EMM Unfortunately the video in your last comment doesn't play for me. If you could re-upload it, that would be very helpful.

Just to avoid any misunderstandings, is it correct that you're doing the following steps?

Open Blender
Open the keymap user preferences Edit > Preferences > Keymap
Select a keymap and then press Export (First error appears on export)
Click on Import, select the previously exported file (Second error appears)

Was the first keymap that resulted in the export/import error created by editing one of the built-in keymaps through Edit > Preferences > Keymap? Is the EMM.py or .py (from your original report) the keymap what you get after exporting and you're trying to import this file again? Is keymap.py what you get after attempting to import EMM.py or .py?

@Guai_Wo_Ge_EMM Unfortunately the video in your last comment doesn't play for me. If you could re-upload it, that would be very helpful. Just to avoid any misunderstandings, is it correct that you're doing the following steps? - Open Blender - Open the keymap user preferences *Edit > Preferences > Keymap* - Select a keymap and then press *Export* (First error appears on export) - Click on *Import*, select the previously exported file (Second error appears) Was the first keymap that resulted in the export/import error created by editing one of the built-in keymaps through *Edit > Preferences > Keymap*? Is the `EMM.py` or `.py` (from your original report) the keymap what you get after exporting and you're trying to import this file again? Is `keymap.py` what you get after attempting to import `EMM.py` or `.py`?

Yevgeny Makarov commented

2020-11-01 14:22:38 +01:00

Added subscriber: @jenkm

Yevgeny Makarov commented

2020-11-01 14:22:38 +01:00

I don't know what the "Blender-PME" addon is, but you use non-English names in it, I think so.

It could be something like this {86c4ca71ef18}.

    ("wm.pme_user_pie_menu_call",
     {"type": 'SPACE', "value": 'PRESS'},
     {"properties":
      [("pie_menu_name", '±à¼Ä£Ê½-Íø¸ñÊý¾Ý'),
       ("invoke_mode", 'HOTKEY'),
       ("keymap", '3D View'),
       ],
      },
     ),

I don't know what the "Blender-PME" addon is, but you use non-English names in it, I think so. It could be something like this {86c4ca71ef18}. ``` ("wm.pme_user_pie_menu_call", {"type": 'SPACE', "value": 'PRESS'}, {"properties": [("pie_menu_name", '±à¼Ä£Ê½-Íø¸ñÊý¾Ý'), ("invoke_mode", 'HOTKEY'), ("keymap", '3D View'), ], }, ), ```

EMM commented

2020-11-01 14:26:35 +01:00

This is a re-uploaded video {F9151429} , but I can't seem to watch it either, so I've changed the .mp4 file to .py so you can download it, change the extension and you'll be able to view it!

这是重新上传的视频 {F9151429} ,但是我好像也看不了,所以我把这个.mp4文件改为.py,这样您就可以下载下来,更改一下后缀名称就能查看了

``` This is a re-uploaded video {F9151429} , but I can't seem to watch it either, so I've changed the .mp4 file to .py so you can download it, change the extension and you'll be able to view it! ``` ``` 这是重新上传的视频 {F9151429} ,但是我好像也看不了,所以我把这个.mp4文件改为.py,这样您就可以下载下来,更改一下后缀名称就能查看了

EMM commented

2020-11-01 14:32:12 +01:00

In #82292#1045707, @jenkm wrote:
I don't know what the "Blender-PME" addon is, but you use non-English names in it, I think so.

Yes, I'm using a PIE plugin, it can use custom UI.
Its forum address is this:https://blenderartists.org/t/pie-menu-editor-1-18-0/662456/3680

是的,我正在使用一个PIE插件,它能使用自定义UI.是这个原因吗
它的论坛地址是这个:https://blenderartists.org/t/pie-menu-editor-1-18-0/662456/3680

> In #82292#1045707, @jenkm wrote: > I don't know what the "Blender-PME" addon is, but you use non-English names in it, I think so. > Yes, I'm using a PIE plugin, it can use custom UI. Its forum address is this:https://blenderartists.org/t/pie-menu-editor-1-18-0/662456/3680 是的,我正在使用一个PIE插件,它能使用自定义UI.是这个原因吗它的论坛地址是这个:https://blenderartists.org/t/pie-menu-editor-1-18-0/662456/3680

Robert Guetzkow commented

2020-11-01 14:48:57 +01:00

I can reproduce the import error by attempting to import EMM.py. This is caused by the pie menu entries as @jenkm said and as I've noted in my first comment the encoding for these characters is not in UTF-8 but seem to be GB2312. ~~It appears that the Blender-PME add-on doesn't handled encoding properly and stores non UTF-8 characters in the keymap. We can't provide support for the paid third-party add-on,~~ however Blender shouldn't let an add-on store invalid characters in the keymap either.

I'll try to reproduce this issue through the Python API.

Edit: It's Blender's fault.

I can reproduce the import error by attempting to import `EMM.py`. This is caused by the pie menu entries as @jenkm said and as I've noted in my first comment the encoding for these characters is not in UTF-8 but seem to be GB2312. ~~It appears that the Blender-PME add-on doesn't handled encoding properly and stores non UTF-8 characters in the keymap. We can't provide support for the paid third-party add-on,~~ however Blender shouldn't let an add-on store invalid characters in the keymap either. I'll try to reproduce this issue through the Python API. Edit: It's Blender's fault.

Robert Guetzkow commented

2020-11-01 15:06:02 +01:00

Changed status from 'Needs User Info' to: 'Needs Developer To Reproduce'

Robert Guetzkow commented

2020-11-01 15:15:49 +01:00

@Guai_Wo_Ge_EMM As a workaround I've edited the EMM.py file which can now be imported. The Chinese characters have been replaced with the (poorly machine translated) English words and it's been saved with UTF-8 encoding.

EMM_edit.py

@Guai_Wo_Ge_EMM As a workaround I've edited the `EMM.py` file which can now be imported. The Chinese characters have been replaced with the (poorly machine translated) English words and it's been saved with UTF-8 encoding. [EMM_edit.py](https://archive.blender.org/developer/F9151880/EMM_edit.py)

EMM commented

2020-11-01 15:48:27 +01:00

Thank you, the file you sent can be imported, thank you very much, in addition to this bug I have feedback to the plug-in developers there, should be resolved!
谢谢您,您发的文件可以导入了,十分感谢,另外这个BUG我已经反馈到插件开发者那里了,应该会得到解决

Thank you, the file you sent can be imported, thank you very much, in addition to this bug I have feedback to the plug-in developers there, should be resolved! 谢谢您,您发的文件可以导入了,十分感谢,另外这个BUG我已经反馈到插件开发者那里了,应该会得到解决

Robert Guetzkow commented

2020-11-02 11:32:45 +01:00

I've tried to intentionally break the keymap through the Python API, but haven't been successful so far. Will continue to investigate.

Robert Guetzkow commented

2020-11-03 14:50:24 +01:00

It seems that the problem is caused when an operator has a string property, there is a shortcut registered for it in the keymap and the shortcut uses a value for the operators string property that is not UTF-8 encoded. However, I can't get a keymap to export after assigning e.g. Chinese characters to the property in the keymap preferences. It gives the following exception:

location: <unknown location>:-1
Traceback (most recent call last):
  File "Z:\02_software\blender-2.81a-windows64\2.81\scripts\startup\bl_operators\userpref.py", line 259, in execute
    all_keymaps=self.all,
  File "Z:\02_software\blender-2.81a-windows64\2.81\scripts\modules\bl_keymap_utils\io.py", line 194, in keyconfig_export_as_data
    fw(kmi_data)
  File "Z:\02_software\blender-2.81a-windows64\2.81\python\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 37-38: character maps to <undefined>

As of now I cannot reproduce the exact error from the original bug report with my current system configuration, but I have an idea. Since the error references cp1252.py Blender always seems to be using the system locale to determine the encoding. Reproducing the exact error from the original report may be possible with a different system locale configuration, which then presumably fails to import it later on. I would guess that in order to reproduce this bug you would have to set Windows' system locale to Simplified Chinese, which then results in code page 936 being used. This would then have GB2312 encoding applied to the input, which is what I've seen in the keymap included in the bug report.

If anyone wants to test this and investigate further, below is a script that implements a simple operator which is added to the keymap when registered.

operator-to-reproduce-bug.py

It seems that the problem is caused when an operator has a string property, there is a shortcut registered for it in the keymap and the shortcut uses a value for the operators string property that is not UTF-8 encoded. However, I can't get a keymap to export after assigning e.g. Chinese characters to the property in the keymap preferences. It gives the following exception: ``` location: <unknown location>:-1 Traceback (most recent call last): File "Z:\02_software\blender-2.81a-windows64\2.81\scripts\startup\bl_operators\userpref.py", line 259, in execute all_keymaps=self.all, File "Z:\02_software\blender-2.81a-windows64\2.81\scripts\modules\bl_keymap_utils\io.py", line 194, in keyconfig_export_as_data fw(kmi_data) File "Z:\02_software\blender-2.81a-windows64\2.81\python\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 37-38: character maps to <undefined> ``` As of now I cannot reproduce the exact error from the original bug report with my current system configuration, but I have an idea. Since the error references `cp1252.py` Blender always seems to be using the system locale to determine the encoding. Reproducing the exact error from the original report may be possible with a different system locale configuration, which then presumably fails to import it later on. I would guess that in order to reproduce this bug you would have to set Windows' system locale to Simplified Chinese, which then results in code page 936 being used. This would then have GB2312 encoding applied to the input, which is what I've seen in the keymap included in the bug report. If anyone wants to test this and investigate further, below is a script that implements a simple operator which is added to the keymap when registered. [operator-to-reproduce-bug.py](https://archive.blender.org/developer/F9181126/operator-to-reproduce-bug.py)

Robert Guetzkow commented

2020-11-03 15:13:22 +01:00

I think I found the issue. The code page error in my last comment means that Blender isn't using UTF-8 for the export on my system either. The function keyconfig_export_as_data that writes the keymap doesn't set the encoding to UTF-8. The encoding should be explicitly set in Python's open() function.

Currently:

    with open(filepath, "w") as fh:

Possible fix:

    with open(filepath, "w", encoding="utf-8") as fh:

Edit: @jenkm was spot on with his comment.

I think I found the issue. The code page error in my last comment means that Blender isn't using UTF-8 for the export on my system either. The function `keyconfig_export_as_data` that writes the keymap doesn't set the encoding to UTF-8. The encoding should be explicitly set in Python's `open()` function. Currently: ``` with open(filepath, "w") as fh: ``` Possible fix: ``` with open(filepath, "w", encoding="utf-8") as fh: ``` Edit: @jenkm was spot on with his comment.

Yevgeny Makarov commented

2020-11-03 15:16:11 +01:00

@rjg

In #82292#1045707, @jenkm wrote:
It could be something like this {86c4ca71ef18}.

:)

@rjg > In #82292#1045707, @jenkm wrote: > It could be something like this {86c4ca71ef18}. :)

Robert Guetzkow commented

2020-11-03 15:20:59 +01:00

@jenkm Yes, I know. Would've found/verified the problem quicker if I checked keyconfig_export_as_data or grepped open( first, instead of attempting to reproduce the reported error condition.

@jenkm Yes, I know. Would've found/verified the problem quicker if I checked `keyconfig_export_as_data` or grepped `open(` first, instead of attempting to reproduce the reported error condition.

Robert Guetzkow commented

2020-11-03 15:21:17 +01:00

Changed status from 'Needs Developer To Reproduce' to: 'Confirmed'

Robert Guetzkow changed title from ~~Cannot import the same version of exported shortcuts 无法导入同一个版本导出的快捷键~~ to Keymap is not exported with the correct encoding

2020-11-03 15:25:08 +01:00

Robert Guetzkow self-assigned this 2020-11-03 15:25:27 +01:00

blender-admin commented

2020-11-04 10:45:36 +01:00

This issue was referenced by 4988f5f39c

This issue was referenced by 4988f5f39ca2819516a487019008fb7e0dc28b28

blender-admin commented

2020-11-04 10:45:36 +01:00

This issue was referenced by f7320c3bf1

This issue was referenced by f7320c3bf159f14f0c39f88b2302a6e64e7eef76

Robert Guetzkow commented

2020-11-04 13:23:25 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Robert Guetzkow closed this issue

2020-11-04 13:23:25 +01:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Keymap is not exported with the correct encoding #82292