The OptiX implementation wasn't trying to allocate memory on the host if device allocation failed, while the CUDA implementation does.
Looks like many changes, but it really only copies most of the memory allocation code from device_cuda over to device_optix to remedy this. The code in device_cuda is proven to work, so is safe to use.
Eventually this should be moved out into a common file and shared between device_cuda and device_optix, but for now this is the quickest solution to fix in 2.81.
I checked this against a big range of scene and encountered no issues, plus the scenes that failed previously now work (e.g. the spring splash screen on a small GPU like a RTX 2060).