Cycles HIP error with image textures on Linux and RDNA1 #97591

New Issue

Willie Sippel · 2022-04-24T17:26:55+02:00

Willie Sippel commented

2022-04-24 17:26:55 +02:00

System Information
Operating system: Arch Linux
Graphics card: AMD 5700XT

Blender Version
Broken: Blender 3.2 alpha c486da0238
Worked: never

Short description of error
Trying to render a scene with HIP support enabled crashes Blender during the "Updating images" stage. Scenes without image textures (eg the default cube, but also elaborate scenes with complex procedural shaders) render perfectly fine. Tested with the open source ROCm stack from rocm-arch and the official binaries provided by AMD, on kernels 5.15.35-lts, 5.17.4-zen1 and 5.17.4-xanmod1. Also tested with the first HIP enabled nightly build, same result. As a user on the Blender forum reported success on Xorg, I tested on both Gnome/ Xorg and Gnome/ Wayland. Looking through the thread again, it seems all successful reports come from users with RDNA2 GPUs, while the only other failure was reported by another 5700XT owner. The last few lines from the console log:

:3:hip_texture.cpp          :1453: 15928128518 us: 33981: [tid:0x7f0decdff640] hipTexObjectCreate ( 0x7f0,deb,8ed,2e8, 0x7f0,dec,db9,140, 0x7f0,dec,db9,0d0, char array:<null> )
:4:rocdevice.cpp            :2034: 15928128655 us: 33981: [tid:0x7f0decdff640] Allocate hsa device memory 0x7f0c9ca00000, size 0x169000
:3:rocdevice.cpp            :2073: 15928128661 us: 33981: [tid:0x7f0decdff640] device=0x7f0df5092000, freeMem_ = 0xfed20000
:4:rocdevice.cpp            :1899: 15928128724 us: 33981: [tid:0x7f0decdff640] Allocate hsa host memory 0x7f0e10c1e000, size 0x110
Writing: /tmp/bmw27_gpu.crash.txt
fish: Job 1, 'AMD_LOG_LEVEL=4 ./blender' terminated by signal SIGSEGV (Address boundary error)

And the crash log:

# Blender 3.2.0, Commit date: 2022-04-23 13:09, Hash c486da0238bd

# backtrace
./blender(BLI_system_backtrace+0x20) [0xb605b00]
./blender() [0x117952a]
/usr/lib/libc.so.6(+0x42560) [0x7f0e24064560]
/opt/rocm/hip/lib/libamdhip64.so(+0x1e8b3d) [0x7f0dda9e8b3d]
/opt/rocm/hip/lib/libamdhip64.so(hipTexObjectCreate+0x8e1) [0x7f0dda9f1901]
./blender(_ZN3ccl9HIPDevice9tex_allocERNS_14device_textureE+0x6b5) [0x2d4d235]
./blender(_ZN3ccl12ImageManager17device_load_imageEPNS_6DeviceEPNS_5SceneEiPNS_8ProgressE+0x392) [0x3464122]
./blender() [0x84a86e5]
./blender() [0x16759b5]
./blender() [0x1675c6b]
./blender() [0x1662987]
./blender() [0x166f6a0]
./blender() [0x16716dc]
./blender() [0x16718d9]
/usr/lib/libc.so.6(+0x8d5c2) [0x7f0e240af5c2]
/usr/lib/libc.so.6(clone+0x44) [0x7f0e24134584]

# Python backtrace

Exact steps for others to reproduce the error
Render anything with an image texture with HIP and GPU compute enabled. I used the BMW benchmark to create the logs.

**System Information** Operating system: Arch Linux Graphics card: AMD 5700XT **Blender Version** Broken: Blender 3.2 alpha c486da0238bd Worked: never **Short description of error** Trying to render a scene with HIP support enabled crashes Blender during the "Updating images" stage. Scenes without image textures (eg the default cube, but also elaborate scenes with complex procedural shaders) render perfectly fine. Tested with the open source ROCm stack from rocm-arch and the official binaries provided by AMD, on kernels 5.15.35-lts, 5.17.4-zen1 and 5.17.4-xanmod1. Also tested with the first HIP enabled nightly build, same result. As a user on the Blender forum reported success on Xorg, I tested on both Gnome/ Xorg and Gnome/ Wayland. Looking through the thread again, it seems all successful reports come from users with RDNA2 GPUs, while the only other failure was reported by another 5700XT owner. The last few lines from the console log: ``` :3:hip_texture.cpp :1453: 15928128518 us: 33981: [tid:0x7f0decdff640] hipTexObjectCreate ( 0x7f0,deb,8ed,2e8, 0x7f0,dec,db9,140, 0x7f0,dec,db9,0d0, char array:<null> ) :4:rocdevice.cpp :2034: 15928128655 us: 33981: [tid:0x7f0decdff640] Allocate hsa device memory 0x7f0c9ca00000, size 0x169000 :3:rocdevice.cpp :2073: 15928128661 us: 33981: [tid:0x7f0decdff640] device=0x7f0df5092000, freeMem_ = 0xfed20000 :4:rocdevice.cpp :1899: 15928128724 us: 33981: [tid:0x7f0decdff640] Allocate hsa host memory 0x7f0e10c1e000, size 0x110 Writing: /tmp/bmw27_gpu.crash.txt fish: Job 1, 'AMD_LOG_LEVEL=4 ./blender' terminated by signal SIGSEGV (Address boundary error) ``` And the crash log: ``` # Blender 3.2.0, Commit date: 2022-04-23 13:09, Hash c486da0238bd # backtrace ./blender(BLI_system_backtrace+0x20) [0xb605b00] ./blender() [0x117952a] /usr/lib/libc.so.6(+0x42560) [0x7f0e24064560] /opt/rocm/hip/lib/libamdhip64.so(+0x1e8b3d) [0x7f0dda9e8b3d] /opt/rocm/hip/lib/libamdhip64.so(hipTexObjectCreate+0x8e1) [0x7f0dda9f1901] ./blender(_ZN3ccl9HIPDevice9tex_allocERNS_14device_textureE+0x6b5) [0x2d4d235] ./blender(_ZN3ccl12ImageManager17device_load_imageEPNS_6DeviceEPNS_5SceneEiPNS_8ProgressE+0x392) [0x3464122] ./blender() [0x84a86e5] ./blender() [0x16759b5] ./blender() [0x1675c6b] ./blender() [0x1662987] ./blender() [0x166f6a0] ./blender() [0x16716dc] ./blender() [0x16718d9] /usr/lib/libc.so.6(+0x8d5c2) [0x7f0e240af5c2] /usr/lib/libc.so.6(clone+0x44) [0x7f0e24134584] # Python backtrace ``` **Exact steps for others to reproduce the error** Render anything with an image texture with HIP and GPU compute enabled. I used the BMW benchmark to create the logs.

Willie Sippel commented

2022-04-24 17:26:55 +02:00

Added subscriber: @wsippel

blender-admin commented

2022-04-24 17:26:55 +02:00

#100711 was marked as duplicate of this issue

blender-admin commented

2022-04-24 17:26:55 +02:00

#98900 was marked as duplicate of this issue

blender-admin commented

2022-04-24 17:26:55 +02:00

#98859 was marked as duplicate of this issue

Richard Antalik commented

2022-04-26 03:19:59 +02:00

Added subscribers: @JacquesLucke, @Jeroen-Bakker, @iss

Richard Antalik commented

2022-04-26 03:19:59 +02:00

@Jeroen-Bakker, @JacquesLucke Can you reproduce? According to HW list you have access to this GPU.

George Sofianos commented

2022-04-28 20:54:34 +02:00

Added subscriber: @Luciddream

Andreas T. commented

2022-04-29 12:11:52 +02:00

Added subscriber: @tschipie

Omar Emara commented

2022-05-11 14:21:55 +02:00

Closed as duplicate of #97997

Omar Emara closed this issue

2022-05-11 14:21:55 +02:00

Brecht Van Lommel commented

2022-05-23 16:46:22 +02:00

Changed status from 'Duplicate' to: 'Confirmed'

Brecht Van Lommel reopened this issue

2022-05-23 16:46:22 +02:00

Brecht Van Lommel commented

2022-05-23 16:46:22 +02:00

Added subscribers: @BrianSavery, @Sayak-Biswas
Removed subscriber: @JacquesLucke

Added subscribers: @BrianSavery, @Sayak-Biswas Removed subscriber: @JacquesLucke

Brecht Van Lommel changed title from ~~Address boundary error rendering with HIP on Linux, maybe RDNA1 specific~~ to Cycles HIP error with image textures on Linux and RDNA1

2022-05-23 16:46:53 +02:00

Brecht Van Lommel commented

2022-05-23 16:48:08 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2022-05-23 16:48:08 +02:00

This report was merged into a bug report about Windows, where a driver update fixed the issue. However the Linux driver is quite different, and I have not seen confirmation yet that it's fixed on Linux.

Marking this as a high priority issue since we really should try to fix this for 3.2.

This report was merged into a bug report about Windows, where a driver update fixed the issue. However the Linux driver is quite different, and I have not seen confirmation yet that it's fixed on Linux. Marking this as a high priority issue since we really should try to fix this for 3.2.

Andreas T. commented

2022-05-23 21:26:15 +02:00

It happens on blender-3.2.0-beta+v32.84e55e3dc251-linux.x86_64-release on Ubuntu 20.04, rocm 5.1.3 and an 5600XT (RDNA1)

:3:hip_texture.cpp          :1453: 1395192224 us: 6547 : [tid:0x7f3489bfe700] hipTexObjectCreate ( 0x7f3,488,fbc,2a8, 0x7f3,489,bb8,1e0, 0x7f3,489,bb8,170, char array:<null> )
:4:rocdevice.cpp            :2035: 1395192555 us: 6547 : [tid:0x7f3489bfe700] Allocate hsa device memory 0x7f330a000000, size 0x1e4000
:3:rocdevice.cpp            :2074: 1395192567 us: 6547 : [tid:0x7f3489bfe700] device=0x7f348d81d000, freeMem_ = 0x7a15c4a8
:4:rocdevice.cpp            :1900: 1395192689 us: 6547 : [tid:0x7f3489bfe700] Allocate hsa host memory 0x7f3490628000, size 0x110
Writing: /tmp/classroom.crash.txt
Segmentation fault (core dumped)

# Blender 3.2.0, Commit date: 2022-05-22 19:10, Hash 84e55e3dc251
Read library:  '/home/andreas/Downloads/classroom/assets/lamps/lamps.blend', '//assets/lamps/lamps.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/chairs/chairs.blend', '//assets/chairs/chairs.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/coatStand/coatStand.blend', '//assets/coatStand/coatStand.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/desks/desks.blend', '//assets/desks/desks.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/dustBin/dustBin.blend', '//assets/dustBin/dustBin.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/radiator/radiator.blend', '//assets/radiator/radiator.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/suitcase/suitcase.blend', '//assets/suitcase/suitcase.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/wallClock/wallClock.blend', '//assets/wallClock/wallClock.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/wastes/wastes.blend', '//assets/wastes/wastes.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/books/books.blend', '//assets/books/books.blend', parent '<direct>'  # Info
Read library:  '/home/andreas/Downloads/classroom/assets/officeSupplies/officeSupplies.blend', '//assets/officeSupplies/officeSupplies.blend', parent '<direct>'  # Info
bpy.context.scene.cycles.device = 'GPU'  # Property

# backtrace
./blender(BLI_system_backtrace+0x20) [0xc3727f0]
./blender() [0x11d83da]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f34cb1e6420]
/opt/rocm/hip/lib/libamdhip64.so(+0x1e8bfd) [0x7f3470721bfd]
/opt/rocm/hip/lib/libamdhip64.so(hipTexObjectCreate+0x8e1) [0x7f347072a9c1]
./blender(_ZN3ccl9HIPDevice9tex_allocERNS_14device_textureE+0x6b5) [0x2eb26e5]
./blender(_ZN3ccl12ImageManager17device_load_imageEPNS_6DeviceEPNS_5SceneEiPNS_8ProgressE+0x392) [0x35d21b2]
./blender() [0x88571b5]
./blender() [0x1793f15]
./blender() [0x17941cb]
./blender() [0x1781207]
./blender() [0x178dc00]
./blender() [0x178fc3c]
./blender() [0x178fe39]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f34cb1da609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f34cabb8133]

# Python backtrace

If you need any specific tests, logs or information please let me know.

[edit}
It doesn't crash with all image textures, the splash screen from v2.81 (The Junk Shop) renders without a problem, although having quite a lot of image textures.

It happens on blender-3.2.0-beta+v32.84e55e3dc251-linux.x86_64-release on Ubuntu 20.04, rocm 5.1.3 and an 5600XT (RDNA1) ``` :3:hip_texture.cpp :1453: 1395192224 us: 6547 : [tid:0x7f3489bfe700] hipTexObjectCreate ( 0x7f3,488,fbc,2a8, 0x7f3,489,bb8,1e0, 0x7f3,489,bb8,170, char array:<null> ) :4:rocdevice.cpp :2035: 1395192555 us: 6547 : [tid:0x7f3489bfe700] Allocate hsa device memory 0x7f330a000000, size 0x1e4000 :3:rocdevice.cpp :2074: 1395192567 us: 6547 : [tid:0x7f3489bfe700] device=0x7f348d81d000, freeMem_ = 0x7a15c4a8 :4:rocdevice.cpp :1900: 1395192689 us: 6547 : [tid:0x7f3489bfe700] Allocate hsa host memory 0x7f3490628000, size 0x110 Writing: /tmp/classroom.crash.txt Segmentation fault (core dumped) ``` ``` # Blender 3.2.0, Commit date: 2022-05-22 19:10, Hash 84e55e3dc251 Read library: '/home/andreas/Downloads/classroom/assets/lamps/lamps.blend', '//assets/lamps/lamps.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/chairs/chairs.blend', '//assets/chairs/chairs.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/coatStand/coatStand.blend', '//assets/coatStand/coatStand.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/desks/desks.blend', '//assets/desks/desks.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/dustBin/dustBin.blend', '//assets/dustBin/dustBin.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/radiator/radiator.blend', '//assets/radiator/radiator.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/suitcase/suitcase.blend', '//assets/suitcase/suitcase.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/wallClock/wallClock.blend', '//assets/wallClock/wallClock.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/wastes/wastes.blend', '//assets/wastes/wastes.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/books/books.blend', '//assets/books/books.blend', parent '<direct>' # Info Read library: '/home/andreas/Downloads/classroom/assets/officeSupplies/officeSupplies.blend', '//assets/officeSupplies/officeSupplies.blend', parent '<direct>' # Info bpy.context.scene.cycles.device = 'GPU' # Property # backtrace ./blender(BLI_system_backtrace+0x20) [0xc3727f0] ./blender() [0x11d83da] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f34cb1e6420] /opt/rocm/hip/lib/libamdhip64.so(+0x1e8bfd) [0x7f3470721bfd] /opt/rocm/hip/lib/libamdhip64.so(hipTexObjectCreate+0x8e1) [0x7f347072a9c1] ./blender(_ZN3ccl9HIPDevice9tex_allocERNS_14device_textureE+0x6b5) [0x2eb26e5] ./blender(_ZN3ccl12ImageManager17device_load_imageEPNS_6DeviceEPNS_5SceneEiPNS_8ProgressE+0x392) [0x35d21b2] ./blender() [0x88571b5] ./blender() [0x1793f15] ./blender() [0x17941cb] ./blender() [0x1781207] ./blender() [0x178dc00] ./blender() [0x178fc3c] ./blender() [0x178fe39] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f34cb1da609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f34cabb8133] # Python backtrace ``` If you need any specific tests, logs or information please let me know. [edit} It doesn't crash with all image textures, the splash screen from v2.81 (The Junk Shop) renders without a problem, although having quite a lot of image textures.

Willie Sippel commented

2022-05-24 00:43:27 +02:00

Interesting. For me, just adding an image texture node to an active material (with no texture actually loaded), and connecting the color output of the texture node to the Principled BSDF color input, crashes Blender immediately if I use Cycles with GPU compute enabled as my viewport renderer.

Adding something else, a Voronoi for example, as color source works just fine, HIP-accelerated raytracing and all.

Interesting. For me, just adding an image texture node to an active material (with no texture actually loaded), and connecting the color output of the texture node to the Principled BSDF color input, crashes Blender immediately if I use Cycles with GPU compute enabled as my viewport renderer. Adding something else, a Voronoi for example, as color source works just fine, HIP-accelerated raytracing and all.

Willie Sippel commented

2022-05-24 01:52:09 +02:00

Can confirm, the Junk Shop scene works. I even triple checked with logging and radeontop to make sure it's really using HIP - it is. No issues with the hardware accelerated Cycles viewport either.

Sayak Biswas commented

2022-05-25 14:48:09 +02:00

I was able to reproduce this issue with a 5700XT + Ubuntu 20.04 + ROCm 5.1.2 with bmw and classroom scenes. I'm looking into this.

Willie Sippel commented

2022-06-04 19:38:55 +02:00

As far as I can tell, the crash happens if a scene uses textures with a horizontal resolution that isn't a multiple of 128. Any random value and any value below 128 I've tested crashes, but 128, 256, 384, 512, 768, 1024, 1536, 1664, 2048 and 4096 all worked perfectly fine. The vertical resolution doesn't matter.

Brecht Van Lommel commented

2022-06-06 17:44:55 +02:00

I looked into adding a workaround for 3.2 that would rescale textures automatically, but it's turning out to be rather complicated and a risky change this close to the release. I think it's more likely we'll wait for a driver fix and release with a warning in the release notes, unless @Sayak-Biswas or @BrianSavery think this is going to take a long time to fix in the driver.

Andrius Lukosevicius commented

2022-06-09 01:37:42 +02:00

Added subscriber: @niobium93

Germano Cavalcante commented

2022-06-14 20:27:58 +02:00

Added subscriber: @Inko

Pratik Borhade commented

2022-06-15 08:10:13 +02:00

Added subscribers: @TeryakiiSauce, @PratikPB2123, @Alaska

Takuro Shoji commented

2022-06-21 10:16:01 +02:00

Added subscriber: @Takuro-Shoji

Caden Mitchell commented

2022-06-24 18:10:45 +02:00

Added subscriber: @Caden-Mitchell

Willie Sippel commented

2022-07-04 08:15:42 +02:00

This issue doesn't seem to be RDNA1 specific after all, a Vega user on the forum reported the same problem: Junk Shop (which uses power-of-two textures) renders just fine, other scenes crash in hipTexObjectCreate.