Page MenuHome

BlenLib/ImageCache : Use after free
Closed, ResolvedPublicBUG

Description

System Information
Operating system: Win10
Graphics card: Meh

Blender Version
Broken: rB339fd8027f0b2d8aaa9efe7ecf0f7af410c14861 (which includes the previous use after free fix done for this code)
Worked: rBb6dd5be2136897bf421027644e519d2ac494022c

Short description of error

The test file from D6267 randomly runs into a use after free , happens rarely, sometimes after 1 run, sometimes after 40.. sooo.... that's.... fun!

ASAN Report

=================================================================
==34220==ERROR: AddressSanitizer: heap-use-after-free on address 0x121fa0342fac at pc 0x7ff75ff1a890 bp 0x0039463faae0 sp 0x0039463faae8
WRITE of size 4 at 0x121fa0342fac thread T0
    #0 0x7ff75ff1a88f in MEM_CacheLimiter<MEM_CacheLimiterHandleCClass>::unmanage K:\BlenderGit\blender\intern\memutil\MEM_CacheLimiter.h:168
    #1 0x7ff75e890214 in moviecache_valfree K:\BlenderGit\blender\source\blender\imbuf\intern\moviecache.c:124
    #2 0x7ff764e46f52 in BLI_ghash_free K:\BlenderGit\blender\source\blender\blenlib\intern\BLI_ghash.c:1012
    #3 0x7ff75e88ebaf in IMB_moviecache_free K:\BlenderGit\blender\source\blender\imbuf\intern\moviecache.c:432
    #4 0x7ff75e157b16 in image_free_cached_frames K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:483
    #5 0x7ff75e14e571 in BKE_image_free_buffers_ex K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:532
    #6 0x7ff760189e3f in ccl::BlenderImageLoader::load_pixels K:\BlenderGit\blender\intern\cycles\blender\image.cpp:149
    #7 0x7ff760a52fb5 in ccl::ImageManager::file_load_image<11,float> K:\BlenderGit\blender\intern\cycles\scene\image.cpp:525
    #8 0x7ff760a4ed94 in ccl::ImageManager::device_load_image K:\BlenderGit\blender\intern\cycles\scene\image.cpp:674
    #9 0x7ff760a596d2 in std::_Func_impl_no_alloc<std::_Binder<std::_Unforced,void (__cdecl ccl::ImageManager::*)(ccl::Device *,ccl::Scene *,int,ccl::Progress *),ccl::ImageManager *,ccl::Device * &,ccl::Scene * &,unsigned __int64 &,ccl::Progress *>,void>::_Do_call k:\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.30.30705\include\functional:822
    #10 0x7ff763c5682c in tbb::internal::function_task<std::function<void __cdecl(void)> >::execute K:\BlenderGit\lib\win64_vc15\tbb\include\tbb\task.h:1059
    #11 0x7ffb85331f55 in tbb::recursive_mutex::scoped_lock::internal_try_acquire+0x28c5 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180011f55)
    #12 0x7ffb85331342 in tbb::recursive_mutex::scoped_lock::internal_try_acquire+0x1cb2 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180011342)
    #13 0x7ff75e9c40c4 in tbb::internal::task_group_base::wait K:\BlenderGit\lib\win64_vc15\tbb\include\tbb\task_group.h:168
    #14 0x7ff763c54686 in ccl::TaskPool::wait_work K:\BlenderGit\blender\intern\cycles\util\task.cpp:44
    #15 0x7ff760a4d5d2 in ccl::ImageManager::device_load_builtin K:\BlenderGit\blender\intern\cycles\scene\image.cpp:863
    #16 0x7ff76017a7cc in ccl::BlenderSession::render K:\BlenderGit\blender\intern\cycles\blender\session.cpp:404
    #17 0x7ff7601706a6 in ccl::render_func K:\BlenderGit\blender\intern\cycles\blender\python.cpp:258
    #18 0x7ffb01806b4c in cfunction_call e:\db21\build\s\vs1564r\build\python\src\external_python\objects\methodobject.c:552
    #19 0x7ffb017bffa6 in _PyObject_MakeTpCall e:\db21\build\s\vs1564r\build\python\src\external_python\objects\call.c:191
    #20 0x7ffb018c7cbe in call_function e:\db21\build\s\vs1564r\build\python\src\external_python\python\ceval.c:5075
    #21 0x7ffb018c47d7 in _PyEval_EvalFrameDefault e:\db21\build\s\vs1564r\build\python\src\external_python\python\ceval.c:3487
    #22 0x7ffb017c0350 in function_code_fastcall e:\db21\build\s\vs1564r\build\python\src\external_python\objects\call.c:330
    #23 0x7ffb017c044d in _PyFunction_Vectorcall e:\db21\build\s\vs1564r\build\python\src\external_python\objects\call.c:367
    #24 0x7ffb018bfc08 in _PyObject_VectorcallTstate e:\db21\build\s\vs1564r\build\python\src\external_python\include\cpython\abstract.h:118
    #25 0x7ffb018c7cbe in call_function e:\db21\build\s\vs1564r\build\python\src\external_python\python\ceval.c:5075
    #26 0x7ffb018c47d7 in _PyEval_EvalFrameDefault e:\db21\build\s\vs1564r\build\python\src\external_python\python\ceval.c:3487
    #27 0x7ffb017c0350 in function_code_fastcall e:\db21\build\s\vs1564r\build\python\src\external_python\objects\call.c:330
    #28 0x7ffb017c044d in _PyFunction_Vectorcall e:\db21\build\s\vs1564r\build\python\src\external_python\objects\call.c:367
    #29 0x7ff75f282288 in bpy_class_call K:\BlenderGit\blender\source\blender\python\intern\bpy_rna.c:8559
    #30 0x7ff75f246af6 in engine_render K:\BlenderGit\blender\source\blender\makesrna\intern\rna_render.c:177
    #31 0x7ff76458fa9b in engine_render_view_layer K:\BlenderGit\blender\source\blender\render\intern\engine.c:910
    #32 0x7ff76458e3e5 in RE_engine_render K:\BlenderGit\blender\source\blender\render\intern\engine.c:1023
    #33 0x7ff76459d37e in do_render_engine K:\BlenderGit\blender\source\blender\render\intern\pipeline.c:1063
    #34 0x7ff76459c9fa in do_render_compositor K:\BlenderGit\blender\source\blender\render\intern\pipeline.c:1192
    #35 0x7ff76459d504 in do_render_full_pipeline K:\BlenderGit\blender\source\blender\render\intern\pipeline.c:1459
    #36 0x7ff764598e38 in RE_RenderAnim K:\BlenderGit\blender\source\blender\render\intern\pipeline.c:2479
    #37 0x7ff75de0901b in arg_handle_render_frame K:\BlenderGit\blender\source\creator\creator_args.c:1600
    #38 0x7ff764e2b17a in BLI_args_parse K:\BlenderGit\blender\source\blender\blenlib\intern\BLI_args.c:311
    #39 0x7ff75de05e20 in main K:\BlenderGit\blender\source\creator\creator.c:524
    #40 0x7ff76504fb8f in __scrt_common_main_seh d:\a01\_work\20\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #41 0x7ffb91aa7033 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #42 0x7ffb91c62650 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180052650)

0x121fa0342fac is located 12 bytes inside of 24-byte region [0x121fa0342fa0,0x121fa0342fb8)
freed by thread T14 here:
    #0 0x7ff76504f26d in operator delete D:\a01\_work\20\s\src\vctools\crt\asan\llvm\compiler-rt\lib\asan\asan_win_delete_scalar_size_thunk.cpp:42
    #1 0x7ff75e890214 in moviecache_valfree K:\BlenderGit\blender\source\blender\imbuf\intern\moviecache.c:124
    #2 0x7ff764e46f52 in BLI_ghash_free K:\BlenderGit\blender\source\blender\blenlib\intern\BLI_ghash.c:1012
    #3 0x7ff75e88ebaf in IMB_moviecache_free K:\BlenderGit\blender\source\blender\imbuf\intern\moviecache.c:432
    #4 0x7ff75e157b16 in image_free_cached_frames K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:483
    #5 0x7ff75e14e571 in BKE_image_free_buffers_ex K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:532
    #6 0x7ff760189e3f in ccl::BlenderImageLoader::load_pixels K:\BlenderGit\blender\intern\cycles\blender\image.cpp:149
    #7 0x7ff760a52fb5 in ccl::ImageManager::file_load_image<11,float> K:\BlenderGit\blender\intern\cycles\scene\image.cpp:525
    #8 0x7ff760a4ed94 in ccl::ImageManager::device_load_image K:\BlenderGit\blender\intern\cycles\scene\image.cpp:674
    #9 0x7ff760a596d2 in std::_Func_impl_no_alloc<std::_Binder<std::_Unforced,void (__cdecl ccl::ImageManager::*)(ccl::Device *,ccl::Scene *,int,ccl::Progress *),ccl::ImageManager *,ccl::Device * &,ccl::Scene * &,unsigned __int64 &,ccl::Progress *>,void>::_Do_call k:\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.30.30705\include\functional:822
    #10 0x7ff763c5682c in tbb::internal::function_task<std::function<void __cdecl(void)> >::execute K:\BlenderGit\lib\win64_vc15\tbb\include\tbb\task.h:1059
    #11 0x7ffb85331f55 in tbb::recursive_mutex::scoped_lock::internal_try_acquire+0x28c5 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180011f55)
    #12 0x7ffb85331342 in tbb::recursive_mutex::scoped_lock::internal_try_acquire+0x1cb2 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180011342)
    #13 0x7ffb85325563 in tbb::interface7::internal::isolate_within_arena+0x3b3 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180005563)
    #14 0x7ffb8532b023 in tbb::task_scheduler_init::terminate+0xbb3 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000b023)
    #15 0x7ffb8532df8b in tbb::thread_bound_filter::try_process_item+0x3bb (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000df8b)
    #16 0x7ffb8532e12c in tbb::thread_bound_filter::try_process_item+0x55c (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000e12c)
    #17 0x7ffb8f7e1bb1 in configthreadlocale+0x91 (C:\WINDOWS\System32\ucrtbase.dll+0x180021bb1)
    #18 0x7ffaf6fdcb43 in _asan_wrap_GlobalSize+0x59f55 (k:\BlenderGit\2022_full\bin\Release\clang_rt.asan_dynamic-x86_64.dll+0x18005cb43)
    #19 0x7ffb91aa7033 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #20 0x7ffb91c62650 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180052650)

previously allocated by thread T14 here:
    #0 0x7ff76504f1fa in operator new D:\a01\_work\20\s\src\vctools\crt\asan\llvm\compiler-rt\lib\asan\asan_win_new_scalar_thunk.cpp:41
    #1 0x7ff75ff19f74 in MEM_CacheLimiter<MEM_CacheLimiterHandleCClass>::insert K:\BlenderGit\blender\intern\memutil\MEM_CacheLimiter.h:159
    #2 0x7ff75ff1a320 in MEM_CacheLimiterCClass::insert K:\BlenderGit\blender\intern\memutil\intern\MEM_CacheLimiterC-Api.cpp:125
    #3 0x7ff75e88fcfa in do_moviecache_put K:\BlenderGit\blender\source\blender\imbuf\intern\moviecache.c:342
    #4 0x7ff75e15c38b in imagecache_put K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:453
    #5 0x7ff75e159812 in image_load_image_file K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:4680
    #6 0x7ff75e15718f in image_acquire_ibuf K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:5171
    #7 0x7ff75e14c811 in BKE_image_acquire_ibuf K:\BlenderGit\blender\source\blender\blenkernel\intern\image.c:5254
    #8 0x7ff75f14e634 in rna_Image_size_get K:\BlenderGit\blender\source\blender\makesrna\intern\rna_image.c:359
    #9 0x7ff7601898b4 in ccl::BlenderImageLoader::load_metadata K:\BlenderGit\blender\intern\cycles\blender\image.cpp:34
    #10 0x7ff760a4e741 in ccl::ImageManager::load_image_metadata K:\BlenderGit\blender\intern\cycles\scene\image.cpp:350
    #11 0x7ff760a4eb26 in ccl::ImageManager::device_load_image K:\BlenderGit\blender\intern\cycles\scene\image.cpp:654
    #12 0x7ff760a596d2 in std::_Func_impl_no_alloc<std::_Binder<std::_Unforced,void (__cdecl ccl::ImageManager::*)(ccl::Device *,ccl::Scene *,int,ccl::Progress *),ccl::ImageManager *,ccl::Device * &,ccl::Scene * &,unsigned __int64 &,ccl::Progress *>,void>::_Do_call k:\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.30.30705\include\functional:822
    #13 0x7ff763c5682c in tbb::internal::function_task<std::function<void __cdecl(void)> >::execute K:\BlenderGit\lib\win64_vc15\tbb\include\tbb\task.h:1059
    #14 0x7ffb85331f55 in tbb::recursive_mutex::scoped_lock::internal_try_acquire+0x28c5 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180011f55)
    #15 0x7ffb85331342 in tbb::recursive_mutex::scoped_lock::internal_try_acquire+0x1cb2 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180011342)
    #16 0x7ffb85325563 in tbb::interface7::internal::isolate_within_arena+0x3b3 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x180005563)
    #17 0x7ffb8532b023 in tbb::task_scheduler_init::terminate+0xbb3 (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000b023)
    #18 0x7ffb8532df8b in tbb::thread_bound_filter::try_process_item+0x3bb (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000df8b)
    #19 0x7ffb8532e12c in tbb::thread_bound_filter::try_process_item+0x55c (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000e12c)
    #20 0x7ffb8f7e1bb1 in configthreadlocale+0x91 (C:\WINDOWS\System32\ucrtbase.dll+0x180021bb1)
    #21 0x7ffaf6fdcb43 in _asan_wrap_GlobalSize+0x59f55 (k:\BlenderGit\2022_full\bin\Release\clang_rt.asan_dynamic-x86_64.dll+0x18005cb43)
    #22 0x7ffb91aa7033 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #23 0x7ffb91c62650 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180052650)

Thread T14 created by T11 here:
    #0 0x7ffaf6fdd968 in _asan_wrap_GlobalSize+0x5ad7a (k:\BlenderGit\2022_full\bin\Release\clang_rt.asan_dynamic-x86_64.dll+0x18005d968)
    #1 0x7ffb8f7e1896 in beginthreadex+0x56 (C:\WINDOWS\System32\ucrtbase.dll+0x180021896)
    #2 0x7ffb8532e2ab in tbb::thread_bound_filter::try_process_item+0x6db (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000e2ab)
    #3 0x7ffb8532df4b in tbb::thread_bound_filter::try_process_item+0x37b (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000df4b)
    #4 0x7ffb8532e12c in tbb::thread_bound_filter::try_process_item+0x55c (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000e12c)
    #5 0x7ffb8f7e1bb1 in configthreadlocale+0x91 (C:\WINDOWS\System32\ucrtbase.dll+0x180021bb1)
    #6 0x7ffaf6fdcb43 in _asan_wrap_GlobalSize+0x59f55 (k:\BlenderGit\2022_full\bin\Release\clang_rt.asan_dynamic-x86_64.dll+0x18005cb43)
    #7 0x7ffb91aa7033 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #8 0x7ffb91c62650 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180052650)

Thread T11 created by T0 here:
    #0 0x7ffaf6fdd968 in _asan_wrap_GlobalSize+0x5ad7a (k:\BlenderGit\2022_full\bin\Release\clang_rt.asan_dynamic-x86_64.dll+0x18005d968)
    #1 0x7ffb8f7e1896 in beginthreadex+0x56 (C:\WINDOWS\System32\ucrtbase.dll+0x180021896)
    #2 0x7ffb8532e2ab in tbb::thread_bound_filter::try_process_item+0x6db (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000e2ab)
    #3 0x7ffb8532a72d in tbb::task_scheduler_init::terminate+0x2bd (k:\BlenderGit\2022_full\bin\Release\tbb.dll+0x18000a72d)
    #4 0x7ff764ea90ac in tbb::task_group::run<Task> K:\BlenderGit\lib\win64_vc15\tbb\include\tbb\task_group.h:208
    #5 0x7ff764ea9983 in tbb_task_pool_run K:\BlenderGit\blender\source\blender\blenlib\intern\task_pool.cc:222
    #6 0x7ff764ea9aba in tbb_task_pool_work_and_wait K:\BlenderGit\blender\source\blender\blenlib\intern\task_pool.cc:240
    #7 0x7ff75e73d100 in blender::deg::deg_evaluate_on_refresh K:\BlenderGit\blender\source\blender\depsgraph\intern\eval\deg_eval.cc:402
    #8 0x7ff75e17158d in scene_graph_update_tagged K:\BlenderGit\blender\source\blender\blenkernel\intern\scene.c:2569
    #9 0x7ff75e58142d in wm_event_do_depsgraph K:\BlenderGit\blender\source\blender\windowmanager\intern\wm_event_system.c:385
    #10 0x7ff75e5620d0 in wm_file_read_post K:\BlenderGit\blender\source\blender\windowmanager\intern\wm_files.c:728
    #11 0x7ff75e56407d in wm_homefile_read_post K:\BlenderGit\blender\source\blender\windowmanager\intern\wm_files.c:1416
    #12 0x7ff75e55c8e1 in WM_init K:\BlenderGit\blender\source\blender\windowmanager\intern\wm_init_exit.c:377
    #13 0x7ff75de05df6 in main K:\BlenderGit\blender\source\creator\creator.c:504
    #14 0x7ff76504fb8f in __scrt_common_main_seh d:\a01\_work\20\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #15 0x7ffb91aa7033 in BaseThreadInitThunk+0x13 (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #16 0x7ffb91c62650 in RtlUserThreadStart+0x20 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180052650)

SUMMARY: AddressSanitizer: heap-use-after-free K:\BlenderGit\blender\intern\memutil\MEM_CacheLimiter.h:168 in MEM_CacheLimiter<MEM_CacheLimiterHandleCClass>::unmanage
Shadow bytes around the buggy address:
  0x045d941685a0: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd
  0x045d941685b0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
  0x045d941685c0: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x045d941685d0: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fd
  0x045d941685e0: fa fa fd fd fd fa fa fa 00 00 00 fa fa fa 00 00
=>0x045d941685f0: 00 00 fa fa fd[fd]fd fa fa fa 00 00 00 00 fa fa
  0x045d94168600: 00 00 00 00 fa fa fd fd fd fa fa fa fd fd fd fd
  0x045d94168610: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
  0x045d94168620: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa
  0x045d94168630: fd fd fd fa fa fa fd fd fd fd fa fa fd fd fd fd
  0x045d94168640: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==34220==ABORTING

Exact steps for others to reproduce the error

Reproducing from the IDE I had no success with, batch file with

:START
blender --factory-startup -noaudio  -b "D6267.blend" -o "//zzz" -F JPEG -x 1 -f 1  >log.txt 2>&1
IF %ERRORLEVEL% EQU 0 goto START

does the trick, given enough patience.

Event Timeline

Jesse Yurkovich (deadpin) changed the task status from Needs Triage to Confirmed.Nov 2 2021, 5:45 AM

Can confirm the ASAN failures here as well. Can you try the following which seems to work well for me: P2564

Some failure of mine were slightly different but they all indicated that it's actually the limitor_lock which was missing around one access to the MEM_CacheLimiter API.

Bastien Montagne (mont29) changed the subtype of this task from "Report" to "Bug".Nov 2 2021, 9:19 AM
Bastien Montagne (mont29) moved this task from Backlog to Blender 3.0 on the Core board.

From a quick look fix proposed in

1diff --git a/source/blender/imbuf/intern/moviecache.c b/source/blender/imbuf/intern/moviecache.c
2index 6cc1932eff6..3a45f68cecf 100644
3--- a/source/blender/imbuf/intern/moviecache.c
4+++ b/source/blender/imbuf/intern/moviecache.c
5@@ -121,7 +121,10 @@ static void moviecache_valfree(void *val)
6 PRINT("%s: cache '%s' free item %p buffer %p\n", __func__, cache->name, item, item->ibuf);
7
8 if (item->ibuf) {
9+ BLI_mutex_lock(&limitor_lock);
10 MEM_CacheLimiter_unmanage(item->c_handle);
11+ BLI_mutex_unlock(&limitor_lock);
12+
13 IMB_freeImBuf(item->ibuf);
14 }
15
looks valid to me, but am not so familiar with this area of code... @Sergey Sharybin (sergey) may know better?

@Bastien Montagne (mont29) Think your patch is correct: the limited is shared across multiple movie caches, so we need to guarantee thread-safe access to the limiter.

Jesse Yurkovich (deadpin) closed this task as Resolved.Nov 5 2021, 5:01 AM
Jesse Yurkovich (deadpin) claimed this task.

Committed after confirming with Ray that he was also seeing a good result on his side.