Home
Diffusion Flamenco Worker (master) Browse

flamenco-worker/README.md
d0e484969a71master

Last Change
Raw File

README.md
View Options

# Flamenco Worker
This is the Flamenco Worker implemented in Python 3.
Author: Sybren A. Stüvel <sybren@blender.studio>
## Installation
There are a few ways to install Flamenco Worker:
- If you have a distributable zip file (see [Packaging for distribution](#packaging-for-distribution))
unzip it, then run `pip install flamenco_worker-xxxx.whl`
- If you have a copy of the source files, run `pip install -e .`
## Configuration
Configuration is read from three locations:
- A hard-coded default in the Python source code.
- `flamenco-worker.cfg` in the current working directory.
- `$HOME/.flamenco-worker.cfg`.
When those files do not exist, they are skipped (i.e. this is not an error). They
should be in INI format, as specified by the
[configparser documentation](https://docs.python.org/3/library/configparser.html)
### Configuration contents:
All configuration keys should be placed in the `[flamenco-worker]` section of the
config files.
- `manager_url`: Flamenco Manager URL.
- `worker_id`: ID of the worker, handed out by the Manager upon registration (see
Registration below) and used for authentication with the Manager.
- `worker_secret`: Secret key of the worker, given to the Manager upon registration
and authentication.
- `job_types`: Space-separated list of job types this worker may execute.
- `task_update_queue_db`: filename of the SQLite3 database holding the queue of task
updates to be sent to the Master.
### TODO
- Certain settings are currently only settable by editing constants in the Python source code.
It might be nice to read them from the config file too, at some point.
- Update worker address in MongoDB when communicating with it.
## Invocation
Install using `pip install -e .` for development, or `setup.py install` for production.
This creates a command `flamenco-worker`, which can be run with `--help` to obtain
a list of possible CLI arguments.
## Registration
If the configuration file does not contain both a `worker_id` and `worker_secret`, at startup
the worker will attempt to register itself at the Master.
Once registered via a POST to the manager's `/register-worker` endpoint, the `worker_id` and
`worker_secret` will be written to `$HOME/.flamenco-worker.cfg`
## Task fetch & execution
1. A task is obtained by the FlamencoWorker from the manager via a POST to its `/task` endpoint.
If this fails (for example due to a connection error), the worker will retry every few seconds
until a task fetch is succesful.
2. The task is given to a TaskRunner object.
3. The TaskRunner object iterates over the commands and executes them.
4. At any time, the FlamencoWorker can be called upon to register activities and log lines,
and forward them to the Manager. These updates are queued in a SQLite3 database, such that
task execution isn't interrupted when the Manager cannot be reached.
5. A separate coroutine of TaskUpdateQueue fetches updates from the queue, and forwards them to
the Master, using a POST to its `/tasks/{task-id}/update` endpoint.
**TODO:** the response to this endpoint may indicate a request to abort the currently running
task. This should be implemented.
## Shutdown
Pressing [CTRL]+[C] will cause a clean shutdown of the worker.
If there is a task currently running, it will be aborted without changing its status. Any pending task updates are sent to the Manager, and then the Manager's `/sign-off` URL is
POSTed to, to indicate a clean shutdown of the worker. Any active task that is still
assigned to the worker is given status "claimed-by-manager" so that it can be re-activated
by another worker.
## Systemd integration
To run Flamenco Worker as a systemd-managed service, copy `flamenco-worker.service` to
`/etc/systemd/system/flamenco-worker.service`, then run `systemctl daemon-reload`.
After installation of this service, `systemctl {start,stop,status,restart} flamenco-worker`
commands can be used to manage it. To ensure that the Flamenco Worker starts at system boot,
use `systemctl enable flamenco-worker`.
## Signals
Flamenco Worker responds to the following POSIX signals:
- `SIGINT`, `SIGTERM`: performs a clean shutdown, as described in the Shutdown section above.
- `SIGUSR1`: logs the currently scheduled asyncio tasks.
## Packaging for distribution
Run `python setup.py zip` to create a distributable zip file. It contains the Wheel,
example configuration file, this README.md, license information and system integration
files.