Home
Diffusion Flamenco Worker (windows-fixes) Browse

flamenco-worker/README.md
c9234a2c13a7windows-fixes

Last Change
Raw File

README.md
View Options

# Flamenco Worker
This is the Flamenco Worker implemented in Python 3.
Author: Sybren A. Stüvel <sybren@blender.studio>
## Installation
Before you begin, make sure you have Flamenco Manager up and running.
There are two ways to install Flamenco Worker:
- If you have a distributable zip file (see [Packaging for distribution](#packaging-for-distribution))
unzip it, then run `pip3 install flamenco_worker-xxxx.whl`
- If you have a copy of the source files, run `pip3 install -e .`
## Configuration
Configuration is read from three locations:
- A hard-coded default in the Python source code.
- `flamenco-worker.cfg` in the current working directory, i.e. the directory that your shell
is in when you invoke the `flamenco-worker` command.
- `$HOME/.flamenco-worker.cfg`; this file is optional.
The configuration files should be in INI format, as specified by the
[configparser documentation](https://docs.python.org/3/library/configparser.html)
### Configuration contents:
All configuration keys should be placed in the `[flamenco-worker]` section of the
config files. At least take a look at:
- `manager_url`: Flamenco Manager URL.
- `task_types`: Space-separated list of task types this worker may execute.
- `task_update_queue_db`: filename of the SQLite3 database holding the queue of task
updates to be sent to the Master.
These configuration keys are also required, but are created automatically in
`$HOME/.flamenco-worker.cfg` when they don't exist yet:
- `worker_id`: ID of the worker, handed out by the Manager upon registration (see
Registration below) and used for authentication with the Manager.
- `worker_secret`: Secret key of the worker, generated by the Worker and given to the
Manager upon registration and authentication.
### TODO
- Certain settings are currently only settable by editing constants in the Python source code.
It might be nice to read them from the config file too, at some point.
## Invocation
Install using `pip3 install -e .` for development, or `python3 setup.py install` for production.
This creates a command `flamenco-worker`, which can be run with `--help` to obtain
a list of possible CLI arguments.
## Registration
If the configuration file does not contain both a `worker_id` and `worker_secret`, at startup
the worker will attempt to register itself at the Master.
Once registered via a POST to the manager's `/register-worker` endpoint, the `worker_id` and
`worker_secret` will be written to `$HOME/.flamenco-worker.cfg`
## Task fetch & execution
1. A task is obtained by the FlamencoWorker from the manager via a POST to its `/task` endpoint.
If this fails (for example due to a connection error), the worker will retry every few seconds
until a task fetch is succesful.
2. The task is given to a TaskRunner object.
3. The TaskRunner object iterates over the commands and executes them.
4. At any time, the FlamencoWorker can be called upon to register activities and log lines,
and forward them to the Manager. These updates are queued in a SQLite3 database, such that
task execution isn't interrupted when the Manager cannot be reached.
5. A separate coroutine of TaskUpdateQueue fetches updates from the queue, and forwards them to
the Master, using a POST to its `/tasks/{task-id}/update` endpoint.
**TODO:** the response to this endpoint may indicate a request to abort the currently running
task. This should be implemented.
## Shutdown
Pressing [CTRL]+[C] will cause a clean shutdown of the worker.
If there is a task currently running, it will be aborted without changing its status. Any pending task updates are sent to the Manager, and then the Manager's `/sign-off` URL is
POSTed to, to indicate a clean shutdown of the worker. Any active task that is still
assigned to the worker is given status "claimed-by-manager" so that it can be re-activated
by another worker.
## Systemd integration
To run Flamenco Worker as a systemd-managed service, copy `flamenco-worker.service` to
`/etc/systemd/system/flamenco-worker.service`, then run `systemctl daemon-reload`.
After installation of this service, `systemctl {start,stop,status,restart} flamenco-worker`
commands can be used to manage it. To ensure that the Flamenco Worker starts at system boot,
use `systemctl enable flamenco-worker`.
## Signals
Flamenco Worker responds to the following POSIX signals:
- `SIGINT`, `SIGTERM`: performs a clean shutdown, as described in the Shutdown section above.
- `SIGUSR1`: logs the currently scheduled asyncio tasks.
## Packaging for distribution
Run `python setup.py zip` to create a distributable zip file. It contains the Wheel,
example configuration file, this README.md, license information and system integration
files.