The issue was caused by crazy space distortion orientation happening
for subsurf modifier.
Solved by making it so subsurf only deforms the surface but keeps
matrices as-is. This is not fully mathematically correct, but is better
that the fall-back solution which was doing wrong matrices anyway.
Also, this is closer to have subsurf was handled prior to the