Windows 10, Nvidia 1060 6GB, also reproduced on a second Windows 10 system with Nvidia 1070.
1/9 daily build blender-2.80-91a155833e59-win64.zip
Short description of error
Cycles preview (Workspace Rendered mode) of object with displacement goes into a loop allocating memory.
Exact steps for others to reproduce the error
CAUTION: You may lose control of the machine and require a hard system reset! I have reproduced this issue on two separate machines, and on one I was unable to get the Task Manager or CTRL-ALT-DEL to respond after locking up Cycles. (Edit: actually this was probably just because there was another minimized GPU intensive application running as well and the combination of that and Blender looping probably just prevented getting any resources to get at the task manager).
- Open the attached .blend which contains the default box with an adaptive subdiv modifier.
- Switch to Rendered mode in the 3dview. Object edges look a bit rough but otherwise ok.
- In the Render Subdivision properties panel, change the Preview Dicing Rate to 2. Note that this does not trigger a preview refresh (it probably should I think?)
- Click the wrench tab to switch to the Modifiers Properties tab.
- Click on the Dicing Scale, type in 0.5 and hit enter. The preview refreshes correctly with the new settings.
- Now Click the increase (>) button at the right-hand end of the Dicing Scale a few times (about a half second between clicks say). After a couple changes (value up to 0.56 usually) the object in the display will pixelate and the render thread will go 100% CPU busy and will start allocating memory without limit until you kill Blender.
This isn't the only way I've gotten it into this state while messing around with Displacement in this file, but the above sequence is 100% repeatable where otherwise it's pretty random. I've gotten it to go berserk like this while twiddling displacement options including view displacement level, the render Subdivision options, and switching back and forth between the different Workspace display modes (lookdev<->rendered etc.).
It will also sometimes do something else weird, where instead of the smooth, connected displaced surface, it seems to be displacing each face of the subdivided mesh independently. I haven't been able to reproduce this reliably.