This issue was produced by a hack in the sculpt mode code from 2.80 when the sculpt API for connectivity info was not available. The smooth brush was the only brush that needed connectivity info, so there were 3 different smooth functions with the connectivity queries implemented for dyntopo, meshes and grids. The mesh version of smoothing was checking the number of connected faces to a vertex to mask the mesh boundaries, which was not covering all cases and was hardcoded in the smooth function itself.
This patch removes all those legacy functions and unifies all smooth functions into a single one using the new API and the automasking system. In order to achieve this, there were needed some extra changes:
- The smooth brush now does not automasks the boundaries by default, so its default preset needs to be updated to enable automasking
- The mesh boundary info is extracted using BMesh and cached int a bitmap, similar to the disconnected elements IDs. This makes boundary detection work as expected in all cases, solving a lot of known issues with the smooth brush. In multires, this info is extracted and cached only at the base mesh level, so it is much more memory efficient than the previous automasking system.
- In order to keep the brushes responsive as they were before, the automasking system can now skip creating the cache when it is not needed for the requested options. This means that for high poly meshes and simple automasking options the brushes won't lag on start.