I've updated the diff with your suggestions.

I've updated the patch based on your feedback.

This can be closed now I think (as D3679 was merged)

I've updated the code based on your feedback

The problem was the the code didn't go into node groups to search for image textures. I've created a patch that should fix this in https://developer.blender.org/D3679

I've updated this to fix the issue described in this ticket too: https://developer.blender.org/T56525

I've updated the diff with your suggestions. I should have looked at the MOD_displace stuff before... :P

I've managed to get it to render directly to the preview output buffer.

I'm not sure how I would bypass the scene setup and render. The whole preview pipeline seems to be built with the intention to render actual blender scenes.

I've also noticed that the thing that seriously slows down my code is the OSD mesh creation. On the squirrel (17 106 quads) OSD creation takes 4.346086 sec while the rest of my modifier code takes 3.414331. I'll try to multi thread a part of my code. If that goes well then I might push the time down to around 1 sec for the non OSD creation stuff.

A quick update. I've worked with the GSoC student YimingWu so see if my work would be useful with his LANPR solution (and it seems to be!).

@Brecht Van Lommel (brecht) Yes that was it! Thank you.

I've tried to compile the "2.79" tag and it still is broken for me. However if I download the official 2.79 build, it works like expected :S

Hmm, If you guys can reproduce then I don't know what I should do to figure out what is wrong.
I'm using gentoo portage to build blender from git. So the build directory is clean. I haven't made any changes to the blender code either. So I don't know what it could be.

I have uploaded the ported code to: https://github.com/DarkDefender/blender-npr-tess2

Just a quick heads up. I've managed to get the code to run in blender master and with opensubdiv 3.0.x

Campbell requested some examples with characters so I took some from BBB:

Yes, thank you for making blender FFmpeg 3.0 ready!

I guess the solution to this problem will be the new bmesh boolean stuff? I mean there is no point in trying to fix the problem with carve if it's going the get phased out soon anyways, right?

I can attach a screenshot and try earlier versions, but I thought because the new error doesn't seem to be because of doubles, I'll just upload a blend file again to see if you can reproduce it...

Ah, I had multiple problems with the boolean operator it seems... The one I attached were because of doubles. Or at least it seems to be the case as I didn't manage to get it to bug out anymore.