Fix T43865: Cycles: Watertight rendering produces artifacts on a huge plane

The issue was caused by numerical instability whrn having ray origin close to a huge
triangle, which could have aused bad ray distance check.

Watertight Woop intersection isn't really addressing such cases, it's dealing with
small triangles far away from the ray origin instead, so it's a bit tricky yo make
it working reliably.

While we're quite close to the release it's safer to do check in Pleaucker coordinates
if ray close to a huge triangle. Likely this additional check combined with some other
tweaks to the code doesn't cause measurable slowdown in the scenes tested here.

After the release we can play a bit more with this code in order to make it more
stable without Pleucker fallback.

intern/cycles/kernel/geom/geom_triangle_intersect.h

@@ -153,14 +153,16 @@ ccl_device_inline bool triangle_intersect(KernelGlobals *kg,
 	/* Calculate scaled z−coordinates of vertices and use them to calculate
 	 * the hit distance.
 	 */
-	const float Az = Sz * A_kz;
-	const float Bz = Sz * B_kz;
-	const float Cz = Sz * C_kz;
-	const float T = U * Az + V * Bz + W * Cz;
+	const float T = (U * A_kz + V * B_kz + W * C_kz) * Sz;
 
-	if ((xor_signmast(T, sign_mask) < 0.0f) ||
-	    (xor_signmast(T, sign_mask) > isect->t * xor_signmast(det, sign_mask)))
-	{
+	/* Perform "near clipping". */
+	const float abs_T = xor_signmast(T, sign_mask);
+	if(abs_T < 0.0f) {
+		return false;
+	}
+	/* Perform "far clipping". */
+	const float abs_det = xor_signmast(det, sign_mask);
+	if(abs_T > isect->t * abs_det) {
 		return false;
 	}
 
@@ -172,12 +174,37 @@ ccl_device_inline bool triangle_intersect(KernelGlobals *kg,
 	{
 		/* Normalize U, V, W, and T. */
 		const float inv_det = 1.0f / det;
+		const float t = T * inv_det;
+
+#ifdef __INTERSECTION_REFINE__
+		/* TODO(sergey): When intersection refine is enabled ray is being
+		 * pushed by quite small epsilon from the surface, which causes
+		 * numerical issues of watertight Woop intersection check with
+		 * huge triangles.
+		 *
+		 * Here we're working this around by checking distance in Pleucker
+		 * coordinates if intersection is suspiciously close to the point
+		 * in order to eliminate self-shadowing.
+		 *
+		 * Ideally we need to solve this in Woop intersection code but
+		 * it's quite tricky.
+		 */
+		if(UNLIKELY(abs_det > 100000.0f && t < 1e-3f)) {
+			const float3 Ng = cross(A - B, C - A);
+			const float pleucker_den = dot(Ng, dir);
+			const float pleucker_T = dot(A, Ng);
+			if(UNLIKELY(pleucker_T * pleucker_den < 0.0f)) {
+				return false;
+			}
+		}
+#endif
+
 		isect->prim = triAddr;
 		isect->object = object;
 		isect->type = PRIMITIVE_TRIANGLE;
 		isect->u = U * inv_det;
 		isect->v = V * inv_det;
-		isect->t = T * inv_det;
+		isect->t = t;
 		return true;
 	}
 	return false;