Fix sculpt neighbor iterator not taking visibility into account

Sculpting tools are designed to ignore hidden geometry and behave like
hidden geometry does not exist.
When getting the neighbors of a vertex, now this takes into account
hidden geometry to avoid returing neighbors which connected edge is not
visible. This should make corner cases of a lot of tools work properly,
especially when working in low poly meshes when is common to have a
single face loop hidden.

Reviewed By: JacquesLucke

Differential Revision: https://developer.blender.org/D11007

source/blender/blenkernel/BKE_paint.h

@@ -486,7 +486,11 @@ typedef struct SculptSession {
   /* Total number of polys of the base mesh. */
   int totfaces;
   /* Face sets store its visibility in the sign of the integer, using the absolute value as the
-   * Face Set ID. Positive IDs are visible, negative IDs are hidden. */
+   * Face Set ID. Positive IDs are visible, negative IDs are hidden.
+   * The 0 ID is not used by the tools or the visibility system, it is just used when creating new
+   * geometry (the trim tool, for example) to detect which geometry was just added, so it can be
+   * assigned a valid Face Set after creation. Tools are not intended to run with Face Sets IDs set
+   * to 0. */
   int *face_sets;
 
   /* BMesh for dynamic topology sculpting */

source/blender/editors/sculpt_paint/sculpt.c

@@ -780,6 +780,10 @@ static void sculpt_vertex_neighbors_get_faces(SculptSession *ss,
   iter->neighbors = iter->neighbors_fixed;
 
   for (int i = 0; i < ss->pmap[index].count; i++) {
+    if (ss->face_sets[vert_map->indices[i]] < 0) {
+      /* Skip connectivity from hidden faces. */
+      continue;
+    }
     const MPoly *p = &ss->mpoly[vert_map->indices[i]];
     uint f_adj_v[2];
     if (poly_get_adj_loops_from_vert(p, ss->mloop, index, f_adj_v) != -1) {