Manager should detect starvation due to blacklisting #59491

New Issue

Sybren A. Stüvel · 2018-12-17T13:57:17+01:00

Sybren A. Stüvel commented

2018-12-17 13:57:17 +01:00

The solution for #50981 does NOT include starvation detection. In other words, if a job has certain tasks that were failed by all available workers (and thus all workers are blacklisted for this job & task type) there is no detection that this happened. As a result, the job will be stuck in 'active' status without it ever having a chance of being finished.

We should have the Manager regularly inspect queued tasks, to see if there is still at least one worker that is not blacklisted and able to execute them. If not, we can mark those tasks as 'failed' to reflect the actual failure on each worker.

The solution for #50981 does NOT include starvation detection. In other words, if a job has certain tasks that were failed by all available workers (and thus all workers are blacklisted for this job & task type) there is no detection that this happened. As a result, the job will be stuck in 'active' status without it ever having a chance of being finished. We should have the Manager regularly inspect queued tasks, to see if there is still at least one worker that is not blacklisted and able to execute them. If not, we can mark those tasks as 'failed' to reflect the actual failure on each worker.

Sybren A. Stüvel self-assigned this 2018-12-17 13:57:17 +01:00

Sybren A. Stüvel commented

2018-12-17 13:57:17 +01:00

Added subscriber: @dr.sybren