Factory default memory cache limit is 4096 MB (32bit builds) #63981

New Issue

ChristopherAnderssarian · 2019-04-29T12:17:11+02:00

2019-04-29 12:17:11 +02:00

System Information
Operating system: Microsoft Windows 7
GPU: ASUS Strix Radeon Vega 64
CPU: Intel(R) Core(TM) i7-5960X
RAM: 32.0 GB
Full System Information: Here

Blender Version

Broken: 2.80, blender-2.80.0-git.513b71c89ad0-windows32
Worked: 2.79b

Short description of error
The Factory Default memory cache limit is 4GB, but the maximum a user can input is 2GB. This causes a crash.

Exact steps for others to reproduce the error

4096

Start Blender with --debug --factory-startup

The crash

Start Blender with --factory-startup
Load a video in MCE or VSE
Let the video cache
Once over 2GB Blender should crash

Call back:

Mapalloc returns null, fallback to regular malloc: len=8294400 in imb_addrectImBuf, total 3177326980
Calloc returns null: len=8294400 in imb_addrectImBuf, total 3192883820

**System Information** Operating system: Microsoft Windows 7 GPU: ASUS Strix Radeon Vega 64 CPU: Intel(R) Core(TM) i7-5960X RAM: 32.0 GB Full System Information: [Here ](https://developer.blender.org/p/Christopher_Anderssarian/) **Blender Version** Broken: 2.80, `blender-2.80.0-git.513b71c89ad0-windows32` Worked: 2.79b **Short description of error** The Factory Default memory cache limit is 4GB, but the maximum a user can input is 2GB. This causes a crash. **Exact steps for others to reproduce the error** 1. 4096 - Start Blender with `--debug --factory-startup` 2. The crash - Start Blender with `--factory-startup` - Load a video in MCE or VSE - Let the video cache - Once over 2GB Blender should crash Call back: ``` Mapalloc returns null, fallback to regular malloc: len=8294400 in imb_addrectImBuf, total 3177326980 Calloc returns null: len=8294400 in imb_addrectImBuf, total 3192883820 ```

ChristopherAnderssarian commented

2019-04-29 12:17:11 +02:00

Added subscriber: @ChristopherAnderssarian

Philipp Oeser commented

2019-04-29 16:10:46 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2019-04-29 16:10:46 +02:00

Cannot reproduce here (linux fedora 29, 16GB RAM).

This is my RAM starting with a limit of 1GB, then increasing to 4GB, then to 8GB [all in the MCE prefetching frames] -- no crash

Seeing you are using a 32bit build, does this happen with a 64bit build? [are you limited to using 32bit builds?]

Cannot reproduce here (linux fedora 29, 16GB RAM). This is my RAM starting with a limit of 1GB, then increasing to 4GB, then to 8GB [all in the MCE prefetching frames] -- no crash ![#63981.png](https://archive.blender.org/developer/F6991925/T63981.png) Seeing you are using a 32bit build, does this happen with a 64bit build? [are you limited to using 32bit builds?]

ChristopherAnderssarian commented

2019-04-29 16:50:21 +02:00

Hmm, I just tried with blender-2.80-337cac760ba9-linux-glibc224-i686 on my Ubuntu VM and I got a crash...

Did you try with a 32-bit build on your end?

Are you limited to using 32bit builds?

Nope, I choose to test them 'cause otherwise they don't get the same love their bigger sibling gets.

Hmm, I just tried with `blender-2.80-337cac760ba9-linux-glibc224-i686` on my Ubuntu VM and I got a crash... *Did you try with a 32-bit build on your end?* > Are you limited to using 32bit builds? Nope, I choose to test them 'cause otherwise they don't get the same love their bigger sibling gets.

Philipp Oeser commented

2019-04-30 09:04:43 +02:00

Not 100% sure, but if this is not a problem with 64bit builds and only happens on 32bit builds, then I guess this is expected?
(always had the assumption that those cannot allocate more then 2GB per process?)

Or are we talking about the fact that a limit over 2GB shouldnt be allowed on those systems?

Not 100% sure, but if this is not a problem with 64bit builds and only happens on 32bit builds, then I guess this is expected? (always had the assumption that those cannot allocate more then 2GB per process?) Or are we talking about the fact that a limit over 2GB shouldnt be allowed on those systems?

Philipp Oeser changed title from ~~Factory default memory cache limit is 4096 MB~~ to Factory default memory cache limit is 4096 MB (32bit builds)

2019-04-30 09:05:13 +02:00

ChristopherAnderssarian commented

2019-04-30 09:39:09 +02:00

In #63981#668304, @lichtwerk wrote:
Not 100% sure, but if this is not a problem with 64bit builds and only happens on 32bit builds, then I guess this is expected?

It doesn't sound like you've tested it.

The maximum you can have in 2.79 is 1GB
The default in 2.79 is 1GB

The maximum a user can enter in 2.80 is 2GB
The default in 2.80 is 4GB

Or are we talking about the fact that a limit over 2GB shouldn't be allowed on those systems?

Yes

> In #63981#668304, @lichtwerk wrote: > Not 100% sure, but if this is not a problem with 64bit builds and only happens on 32bit builds, then I guess this is expected? It doesn't sound like you've tested it. The maximum you can have in 2.79 is 1GB The default in 2.79 is 1GB The maximum a user can enter in 2.80 is 2GB The default in 2.80 is 4GB > Or are we talking about the fact that a limit over 2GB shouldn't be allowed on those systems? Yes

Sergey Sharybin was assigned by Philipp Oeser

2019-04-30 10:32:52 +02:00

Philipp Oeser commented

2019-04-30 10:32:52 +02:00

Added subscriber: @Sergey

Philipp Oeser commented

2019-04-30 10:32:52 +02:00

Thx for getting back.
Yes, this crashes in 32bit blender.

As you say the range you can put in is sort of covered by max_memory_in_megabytes_int() but the default is above that [in 32bit blender]

@Sergey: should we check for this [use max_memory_in_megabytes_int] in BLO_update_defaults_userpref_blend (where U.memcachelimit default is set?)

Thx for getting back. Yes, this crashes in 32bit blender. As you say the range you can put in is sort of covered by `max_memory_in_megabytes_int()` but the default is above that [in 32bit blender] @Sergey: should we check for this [use max_memory_in_megabytes_int] in `BLO_update_defaults_userpref_blend` (where `U.memcachelimit` default is set?)

Sergey Sharybin commented

2019-04-30 11:09:20 +02:00

Added subscriber: @brecht

Sergey Sharybin commented

2019-04-30 11:09:20 +02:00

Technically, there is a mistake in max_memory_in_megabytes_int(), because 32bit process can address up to 4GB of memory (even on Windows, since we do use /LARGEADDRESSAWARE).

So the reason of a crash is coming from the fact, that some other areas of Blender also allocate memory, so movie cache can not use the entire 4GB, and eventually will crash with out of memory error.

Limiting cache to 2GB for 32bit platforms seems artificial: if one simply opens an image sequence, that almost halves possible memory for a cache (because such usecase could use almost all 4GB).

Also, even with the original 2GB limit, if the geometry used 2GB, and on top of than movie cache wants to use 2GB you'll also get a crash. Similar crash will also happen on 64bit platforms, if the physical amount of memory is less than cache size (or when the memory is spent on geometry).

Not sure, probably is better to use lower value by default, and allow users to increase it higher than 2GB but below 4GB manually? That way we at least avoid a lot of crashes in a default configuration. @brecht?

Technically, there is a mistake in `max_memory_in_megabytes_int()`, because 32bit process can address up to 4GB of memory (even on Windows, since we do use `/LARGEADDRESSAWARE`). So the reason of a crash is coming from the fact, that some other areas of Blender also allocate memory, so movie cache can not use the entire 4GB, and eventually will crash with out of memory error. Limiting cache to 2GB for 32bit platforms seems artificial: if one simply opens an image sequence, that almost halves possible memory for a cache (because such usecase could use almost all 4GB). Also, even with the original 2GB limit, if the geometry used 2GB, and on top of than movie cache wants to use 2GB you'll also get a crash. Similar crash will also happen on 64bit platforms, if the physical amount of memory is less than cache size (or when the memory is spent on geometry). Not sure, probably is better to use lower value by default, and allow users to increase it higher than 2GB but below 4GB manually? That way we at least avoid a lot of crashes in a default configuration. @brecht?

Brecht Van Lommel commented

2019-04-30 11:52:10 +02:00

Decreasing it to 2GB by default on 32 bit seems like a reasonable solution. Or min(system_memory()/2, 4GB).

Decreasing it to 2GB by default on 32 bit seems like a reasonable solution. Or `min(system_memory()/2, 4GB)`.

blender-admin commented

2019-05-17 15:02:12 +02:00

This issue was referenced by fccb42c41f

This issue was referenced by fccb42c41f89d64bf0c7ebd75d315adba4a9a927

Sergey Sharybin commented

2019-05-17 15:10:37 +02:00

Changed status from 'Open' to: 'Resolved'

Sergey Sharybin closed this issue

2019-05-17 15:10:37 +02:00

ChristopherAnderssarian commented

2019-05-17 20:41:21 +02:00

I can still crash Blender with: blender-2.80.0-git.3022dd2b27fd-windows32 just by filling the MCE cache.
New report?

I can still crash Blender with: `blender-2.80.0-git.3022dd2b27fd-windows32` just by filling the MCE cache. New report?

Sergey Sharybin commented

2019-05-17 22:25:21 +02:00

The change only affects default startup and does not have affect when you have user preferences saved already.
Does loading factory settings change the cache limit to 2GB?

The change only affects default startup and does not have affect when you have user preferences saved already. Does loading factory settings change the cache limit to 2GB?

ChristopherAnderssarian commented

2019-05-17 22:40:18 +02:00

In #63981#681692, @Sergey wrote:
Does loading factory settings change the cache limit to 2GB?

Yes it does.
I always test with --factory-startup.
It shows 2048MB in the settings, but goes above that with the prefetch. (once over 3GB it crashes)
If you manually input 2048MB then it's fine, just factory startup doesn't seem to apply the limit...

> In #63981#681692, @Sergey wrote: > Does loading factory settings change the cache limit to 2GB? Yes it does. I always test with `--factory-startup`. It shows 2048MB in the settings, but goes above that with the prefetch. (once over 3GB it crashes) If you manually input 2048MB then it's fine, just factory startup doesn't seem to apply the limit... ![over 3GB.PNG](https://archive.blender.org/developer/F7044197/over_3GB.PNG)

Sergey Sharybin commented

2019-05-18 12:06:02 +02:00

Changed status from 'Resolved' to: 'Open'

Sergey Sharybin reopened this issue

2019-05-18 12:06:02 +02:00

blender-admin commented

2019-05-31 12:06:57 +02:00

This issue was referenced by 77d119a231

This issue was referenced by 77d119a2311133f4cda4e4780486d13d302384a6

Sergey Sharybin commented

2019-05-31 12:09:35 +02:00

Changed status from 'Open' to: 'Resolved'