internet-equity / fate

OS-level periodic execution controller and queue manager

Fate

The operating system-level command scheduler and manager.

TODO: Objectives/justifications; features; installation and quick start-up…​.

Creating tasks

Fate tasks are merely executables available through the file system. Tasks abiding by the framework’s expectations may be added with a minimum of effort.

The contract

TODO

Testing

execute

Any executable may be invoked (with optional arguments) by the Fate execute command:

fate debug execute [options] command [arguments]

The above generates an execution report for use in development and debugging.

Options such as -i|--stdin may be useful to supply measurment parameters to the executable according to the framework’s contract.

run

Once added to Fate configuration, executables become tasks. These may be invoked ad-hoc by the run command:

fate debug run [options] task

The options and output of the run command are similar to those of execute.

Unlike with scheduled tasks, the results of tasks performed by run are not, by default, persisted to file. Either specify option --record to capture these as configured, or option --stdout to capture these at an arbitrary path.

Development

Extending the framework

Set-up

Fate is implemented in Python and the framework’s distribution is managed via Poetry.

Python v3.10 may be supplied by an operating system package manager, by python.org, or by a utility such as pyenv; pyenv is recommended for development but not required.

With Python installed, Poetry may be installed according to its instructions.

Tip	If you are managing your own virtual environment, e.g. via pyenv-virtualenv, then this step may be as simple as pip install poetry. However, this tooling is not required, and Poetry offers its own automated set-up, as well as management of virtual environments.

Finally, from the root directory of a repository clone, the framework may be installed in development mode:

poetry install

Note	Poetry will use any existing, activated virtual environment, or create its own into which dependencies will be installed.

The fate command is now available for use in your development environment.

Important

For simplicity, it will be presumed that fate is available on your PATH. However, this depends upon activation of your virtual environment. A virtual environment under management by Poetry may be activated via sub-shell with: poetry shell Alternatively, any command installed into Poetry’s virtual environment may be executed ad-hoc via the run command: poetry run fate ...

fate's Issues

pathlib.Path.readlink missing in Py38

uncaught exception on first execution (w/o one-line installer)

During an initial run, Fate may crash with an uncaught exception, if it was not installed by the one-line installer.

---

When searching for collisions among "friendly" state directories, Fate must anticipate that the base state directory does not yet exist.

netrics.d: fatal: FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.local/state/netrics'

AttributeError raised by conf-loading descriptor obscured by stack overflow

I tried to netrics run … after pip install rc1 … I got:

Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized

Thread 0x0000ffff7f4031e0 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 302 in wait
  File "/usr/lib/python3.8/queue.py", line 170 in get
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/plumbum/commands/processes.py", line 225 in _timeout_thread_func
  File "/usr/lib/python3.8/threading.py", line 870 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x0000ffff8019e010 (most recent call first):
  File "/usr/lib/python3.8/pathlib.py", line 722 in __str__
  File "/usr/lib/python3.8/pathlib.py", line 729 in __fspath__
  File "/usr/lib/python3.8/pathlib.py", line 1198 in stat
  File "/usr/lib/python3.8/pathlib.py", line 1407 in exists
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 73 in <listcomp>
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 73 in __path__
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/descriptors.py", line 23 in __get__
...
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 100 in _format_
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 104 in _loader_
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 115 in __getdata__
  ...
Aborted (core dumped)

Originally posted by @ggmartins in internet-equity/netrics#21 (comment)

pure Python package initialization

As a user of Fate, I would like to be able to install the library via pip and to be able to initialize system-level features (which might otherwise be handled by my operating system package manager).

For example:

• (default) configuration files
• shell completion
• systemd integration
• man pages

(This is especially important for distributions of Fate which do not "fall back" to built-in configuration files, but rather require that these have been installed to the system.)

uncaught exception when enqueuing secondary cohort

It appears possible for Fate to crash when performing a "midair refueling" – when enqueuing a secondary cohort of tasks to execute – (in response to a relatively long execution time of the primary cohort).

---

The exception is a collections.deque error – deque mutated during iteration – (presumably raised by SchedulingQueue.append and presumably because another SchedulingQueue method which relies on deque iteration does not properly terminate).

resources.files() missing on Python v3.8

I tried to netrics run … after pip install rc1 … I got:

Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized

Thread 0x0000ffff7f4031e0 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 302 in wait
  File "/usr/lib/python3.8/queue.py", line 170 in get
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/plumbum/commands/processes.py", line 225 in _timeout_thread_func
  File "/usr/lib/python3.8/threading.py", line 870 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x0000ffff8019e010 (most recent call first):
  File "/usr/lib/python3.8/pathlib.py", line 722 in __str__
  File "/usr/lib/python3.8/pathlib.py", line 729 in __fspath__
  File "/usr/lib/python3.8/pathlib.py", line 1198 in stat
  File "/usr/lib/python3.8/pathlib.py", line 1407 in exists
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 73 in <listcomp>
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 73 in __path__
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/descriptors.py", line 23 in __get__
...
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 100 in _format_
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 104 in _loader_
  File "/home/ubuntu/netrics/v2/venv/lib/python3.8/site-packages/fate/conf/base/conf.py", line 115 in __getdata__
  ...
Aborted (core dumped)

Originally posted by @ggmartins in internet-equity/netrics#21 (comment)

---

Having applied #9 the error is:

[LoadAttributeError] module 'importlib.resources' has no attribute 'files'

configuration of task execution user

As a configurer of Fate, I should be able to specify the OS user under which scheduled tasks are executed.

In detail:

• Task configuration's user key should allow interpolation, to support e.g.:

  user: "{{ env.TASK_USER }}"

And perhaps:

• User configuration might also support a global default value in the defaults file.

• Though unlikely, also conceivable that I might want introduced new identifiers to indicate an expression which must not be cast to string, say to allow a non-interpolating expression like:

  user: "{~ env.TASK_UID | int ~}"

Note:

• subprocess.Popen(…, user=None, …): string is converted to UID and int is passed through. (And such keywords may be passed through plumbum.machines.local.LocalCommand.run_bg(user=…).)
  • A generic task helper property for such keywords might be named mods_.
  • (plumbum does offer local.as_user() [doc]; however, this appears thread-unsafe. Though this is not a current concern, perhaps better to use run_bg() keywords.)

configuration of result file name (template)

As a user of Fate I would like to be able to configure the names applied to result files.

Currently, I may configure the directory to which they are written, but no more than that.

This should be optionally under my control; and, this would allow me to specify a file extension when this couldn't (or shouldn't) be automatically determined by the software.

---

The template expression of this field should be given standard variables, including default (the default stem) such that only a file extension may be specified, and ext (the extension that would otherwise be applied) such that this may still be automatically determined.

And/or, this likely calls for dynamic specification of path.result as a mapping, explicitly distinguishing the file name from its directory path. Regardless, ext could be optionally specified (statically) on its own.

init-conf command overwrites configuration

When invoked without standard input, or with the --no-prompt flag, the init conf command overwrites any existing configuration, (without prompting).

This is particularly problematic now that the Netrics installer invokes this command, most commonly without standard input. The installer does support upgrades.

The init conf command should differ from the init comp command, or regardless, should not overwrite existing configuration when it cannot prompt the user first.

See also: #15

crash when scheduled command is killed

The Fate runtime may crash when the command it is running is killed by another process.

---

The issue appears to be that Fate does not expect negative exit codes. However, negative exit codes are permitted, such as -X, to indicate that the process was killed by signal X.

Fate should presumably allow negative exit codes and treat these no differently from other exit codes. (See CommandStatus.)

configurable task execution timeout

As a user of Fate, I would like to be able to configure a duration past which task(s)' execution should be interrupted.

By default, this may be None and ignored. However, it should be configurable both globally and per-task. The form of this configuration value may either be a simple integer denoting a number of seconds or (perhaps) a complex string similar to the timeout command (e.g. 1s, 6m, etc.).

When specified, should the duration of a task's execution exceed this value, the task's process should be killed, and task-processing continue as though the process exited otherwise. (Little task-processing may need to change, as the task will indeed have exited, and likely with an exit code appropriate to it having been killed; however, some additional information may be helpful, even just a log message about the timeout/termination.)

---

Presumably, this logic should/must be inserted into ScheduledTask.poll (alias ready), e.g.:

…

if not self._future_.poll() and not self._killed and self.timed_out():
    if not self._terminated:
        self.terminate()
    else:
        self.kill()

if ready := self._future_.poll():
    self._end_time = time.time()
    …

…

With supporting code of the like:

def __init__(…):
    …
    self._start_time = self._end_time = None

def __call__(…):
    …
    self._future_ = future
    self._start_time = time.time()
    …

def terminate(self):
    self._future_.proc._proc.terminate()
    self._terminated = True

def kill(self):
    self._future_.proc._proc.kill()
    self._killed = True

def timed_out(self):
    if self._start_time is None or (timeout := self.timeout_) is None:
        return False

    end_time = self._end_time or time.time()
    return end_time - self._start_time > timeout

---

A log message should perhaps be written when a timeout occurs. And/or, it might be sufficient and most convenient for this to be reflected by the message already logged by ControlCommand.finish_task, though based on a ScheduledTask.stopped() rather than merely on the exit code. E.g.:

status_record = {    
    'status': 'Timeout' if task.stopped() else str(status),
    'exitcode': task.returncode,      
}

where:

def stopped(self):
    return self._terminated or self._killed

init error on Python3.9

See internet-equity/netrics#37.

"no attribute 'is_relative_to'" under Python 3.8

pathlib's Path.is_relative_to wasn't added until v3.9. To support Python 3.8 this must be worked around.

See: internet-equity/netrics#16

configuration of task execution environment variables

As a configurer of Fate, I should be able to specify the environment variables of a scheduled task's execution process.

In detail:

• Environment variables should be configurable at the level of the individual task and globally in the defaults file.

• Environment should be inheritable and this inheritance should be itself configurable. Currently, tasks inherit the environment variables of the scheduler process; and, this should be maintained as the default. Configuration should permit tasks to be isolated from environment inheritance as well.

Further:

• Environment variables might be specified as mappings, under a task key named one of (or perhaps any of): env, environ and/or environment.
• Environment keys and values may include interpolations.
• Inheritance might be specified by task key isolation with string enumeration values:
  • none: (the default) specifies that the task environment merges the scheduler's environment variables, any defaults and any in the task configuration.
  • system: specifies that the task environment merges any defaults with any task configuration – system environment variables are not included.
  • all: specifies that the task environment includes only variables configured for that task.

Note:

• The plumbum env is a frozen copy of the scheduler's env and could be suitable for use. plumbum offers command methods setenv and with_env [doc] (which are thread-safe). Nonetheless, might go thru run_bg() keyword env=… (see #7), at least in so far as this is more straightforward/compact. (Regardless, getting a cleared/non-inheriting env might require that the scheduler constructs its own, or its module's own, LocalMachine (subclass!) to avoid clearing the plumbum.local default/global machine env. …Or, it might be even worse than that.)

unique hash-based scheduling offsets

Fate's scheduling by cron expression with (Jenkins-style) hash-offsets help periodic tasks to not overlap unnecessarily. Task timings are given arbitrary offsets based on a hash of their name. However, across multiple installations, tasks with the same name will still coincide.

In the case of orchestration across multiple devices, this may be undesirable. It could be avoided by simply including a device-unique parameter in the scheduling offset hash – such as that provided by uuid.getnode(). The input for construction of the offset hash would then be of the form: {NAME}.{uuid.getnode()}.

---

This functionality should likely not be assumed by default. Rather, it must be opted in.

This could be a task/default configuration parameter, e.g.:

ping:
  schedule:
    expression: H/5 * * * *
    unique: true

However, likely better – introduce an extension to the cron expression parameter H – U (i.e. "unique"):

ping:
  schedule: U/5 * * * *

In implementation:

• The combination of H and U in a single expression would be a configuration value error

• Unique-hash detected and expression prepared for croniter library by e.g.:

  if expr.startswith('@'):
      hash_unique = False
  else:
      hash_symbol = re.search('H', expr, re.I)
      (expr, hash_unique) = re.subn('U', 'H', expr, flags=re.I)
  
      if hash_symbol and hash_unique:
          raise ConfValueError(…)
  
  …

compat: sched_rr_get_interval not available on darwin

I tried to netrics run on mac … after pip install rc1 … I got:

Traceback (most recent call last):
  File "/Users/gmartins/broadband/netrics/v2/venv/bin/netrics", line 8, in <module>
    sys.exit(main())
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/netrics/__main__.py", line 25, in entrypoint
    hook(
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/fate/cli/root.py", line 18, in entrypoint
    argcmdr.init_package(
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/argcmdr.py", line 120, in init_package
    importlib.import_module(module_info.name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 15, in <module>
    from fate import sched
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/fate/sched/__init__.py", line 1, in <module>
    from .tiered_tenancy import TieredTenancyScheduler  # noqa: F401
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/fate/sched/tiered_tenancy.py", line 13, in <module>
    class TieredTenancyScheduler(TaskScheduler):
  File "/Users/gmartins/broadband/netrics/v2/venv/lib/python3.9/site-packages/fate/sched/tiered_tenancy.py", line 46, in TieredTenancyScheduler
    poll_frequency = os.sched_rr_get_interval(0)
AttributeError: module 'os' has no attribute 'sched_rr_get_interval'

Originally posted by @ggmartins in internet-equity/netrics#21 (comment)