Tested it with yesterdays build. RAM usage stays below 12 GB on a 32 GB system. It no longer fills up RAM and swap file.

This might be the same as T54287, please try with tomorrow's daily build or a new version to verify if it's solved.

Exact steps for others to reproduce the error:
YouTube Video

Under Windows 10 RAM usage stays below 15 GB (around 9-12 if I remember it correctly). In ubuntu 17.10 it never stops filling RAM and swap file. Can't do any further tests though since I'm not at work the next two weeks.

works for me ™

There is no memory leak detected by ASAN, but can confirm it does take a huge amount of RAM very quickly here (just two or three simulated frames at start of baking are enough to fill up my 10GB free RAM…). Then the simulation is a very heavy one, so no idea (or ways to compare with Windows) here. Have the same results with Debian’s 2.79b package, though.

Committed rBc4ef2e2f2ebcd9967aa5d4b283ac7abe51156d51

@Brecht Van Lommel (brecht), thanks for all the helpful reviews! And sure, I'll compile some tests together, as soon as I can.

Looks good to me now.

Ok, one last update... Hopefully...

I have implemented the self collision distance versioning.

Oh, right. That's because the self collision distance is even out of range for the new system. The new system uses actual distance for the self collision, while it was a factor for the old system, so the old default of 0.75 causes a huge amount of BVH overlap results, and makes everything collide. I'll just add versioning to bring that down to the new default.

Enabling Self Collision in the above cloth.blend seems to be extremely slow and the mesh "explodes".

Sorted all the issues I had found.

I found a couple of reliability issues, which I am looking into, and will get sorted ASAP. I'll update this patch as soon as I resolve these problems.

@Brecht Van Lommel (brecht), I forgot to mention, regarding continue_physics.blend not settling, there is another factor at play. The old collision system caused the cloth to be solved twice whenever collisions were found, and it applied both those solves to the cloth, causing it to move twice as fast, so that explains why you might see it settling sooner with the old system. Also, this should explain why more steps are needed to get reliable collisions in the cloth.blend file, as the old solver was effectively using twice as many steps (though the order in which collisions are applied is also a factor here, and I might actually revise the way I'm doing that).

Indeed I had a typo in impulse clamping, that caused it to be ignored. I fixed that now.

Thanks for looking at it, @Brecht Van Lommel (brecht).

This sounds great, will review more closely later.

I changed the way for cache. The error has remained. Only the first frame is baked. The rest of the frames look like a piece of a mesh with a hole.

Cannot confirm any issue with this file on linux, either on 2.79b release or current master… Did you ensure C:/cache/ is writable for you?

@Sergey Sharybin (sergey) you worked on hair recently, maybe you can check on that one? Thanks.

Thanks for your response.

Does this still happen within 2.8? Otherwise there is very little chance this get fixed in 2.7x series now… ;)

Added versioning for the removed booleans (required subversion bump). Renamed "Property Scaling" to "Property Weights", as discussed with @Brecht Van Lommel (brecht) and @Alexander Gavrilov (angavrilov) in IRC. Also renamed the "Collision" panel to "Object Collision", to avoid any impression of it being a superset of the "Self Collision" panel (forgot to do that in the original patch).

I don't think using "Weights" here makes it any easier to understand, as the point of this panel is to vary the value of some properties between vertices by using weights as a tool to interpolate between the base and a new max value. Calling it just "Weights" does nothing towards communicating this.

Seems generally fine.

Non-uniform negative scaling will always be an issue for any process involving matrices… No bug here really, just common known limitation.

I would like to add that for a while, I have been using the new Dependency Graph in Blender, by using this command in the program shortcut: --enable-new-depsgraph

More than a week without reply or activity. Due to the policy of the tracker archiving for until required info/data are provided.

Could not reproduce the crash, but can confirm the weird issue with hairs getting stuck to original position instead of dynamically following the scalp in master. Afaict, in 2.8 everything works fine though, so not sure it’s worth spending more time here, 2.7x is only getting critical fixes nowadays… @Sergey Sharybin (sergey)?

I couldn't spot mistakes in the code, and in some simulation tests it also seems to be working as advertised. So looks good to me, with two minor comments.