Page MenuHome

Job compilation should survive server restarts
Open, Confirmed, HighPublic

Description

Currently any running job compilation is aborted when the server restarts. Since job compilation also can take more than a second, we should probably move it to a background process with Celery. Celery is a bit better in finishing the task before shutting down, and we can have a background monitoring operation and re-create any jobts that are stuck in 'under-construction' status.

This actually happened to a Cloud subscriber (https://cloud.blender.org/flamenco/p-5c73c4680ab12b4c191cf8d0/jobs/5c8673670e2ba003b9e4b33b).

Details

Type
Bug

Event Timeline

Sybren A. Stüvel (sybren) updated the task description. (Show Details)
Sybren A. Stüvel (sybren) triaged this task as Confirmed, High priority.

I think it's need to have ability to remove jobs in 'under-construction' status, anyway, even if it will be fixed.

Nope, deletion of those jobs is impossible for a reason. When the job is in that status, its tasks are being created by Flamenco Server, and nobody else should be touching the job.