Autoscaling Workers

A self-hosted leader can start and stop your workers by running commands you configure in the autoscale section of the leader settings .

The leader runs autoscale.command with the name of a worker and either 1 or 0.

• <command> <worker name> 1 runs when there is work for a worker that has been idle for autoscale.still-up-threshold.
• <command> <worker name> 0 runs when there has been no work for a worker for autoscale.scale-down-timeout.

New work waits for these commands to exit, for every worker. A command still running after autoscale.timeout is stopped.

The leader also runs autoscale.tick.command, or autoscale.command when that is not set, as <command> <worker name> for every worker, every autoscale.tick.interval, unless autoscale.tick.enable is false. Its environment holds these variables and nothing else, so it has no PATH or HOME either:

• NIX_CI_QUEUED_RUNS : Queued runs the worker may take, including builds still waiting on a dependency.
• NIX_CI_DISPATCHABLE_RUNS : Queued runs the worker may take that no dependency is holding back.
• NIX_CI_ONGOING_JOBS : Jobs the worker is running.
• NIX_CI_SECONDS_SINCE_ACTIVITY : Whole seconds since the worker last worked on a job, unset if it never has.
• NIX_CI_SECONDS_SINCE_HEARD : Whole seconds since the worker last contacted the leader, unset if it never has.

Only one leader runs a tick at a time, and a tick waits for every command to exit. A command still running after autoscale.tick.timeout is stopped.

An autoscale command must be idempotent. I.e. use "set the size of the worker pool to X", not "add Y to the size of the worker pool". A tick and a 0 or 1 call can run at the same time for the same worker.

The leader runs each command directly, without a shell, and with nothing from its own environment: the tick gets only the variables above, and the 0 and 1 calls get none at all. Make each one a single executable that brings its own dependencies, for example with writeShellApplication:

{ lib, pkgs, ... }:
let
  autoscale = pkgs.writeShellApplication {
    name = "nix-ci-autoscale";
    runtimeInputs = [ pkgs.curl pkgs.jq ];
    text = ''
      worker_name="$1"
      if [ "$#" -eq 2 ]; then
        echo "Bring $worker_name up (1) or shut it down (0): $2"
      else
        echo "$worker_name could start $NIX_CI_DISPATCHABLE_RUNS runs and is running $NIX_CI_ONGOING_JOBS jobs"
      fi
    '';
  };
in
{
  services.nix-ci.production.leader.config.autoscale.command = lib.getExe autoscale;
}

autoscale: # optional
  command: # required
    # Command to bring a worker up or shut it down, run as <command> <worker name> 1 or <command> <worker name> 0
    <string>
  still-up-threshold: # optional
    # default: 300s
    # How long since a worker last worked on a job before new work wakes it with <command> <worker name> 1
    # any of
    [ <string>
    , <integer> # 64 bit signed integer
    ]
  scale-down-timeout: # optional
    # default: 3600s
    # How long to wait after a worker has done no work before scaling it down
    # any of
    [ <string>
    , <integer> # 64 bit signed integer
    ]
  timeout: # optional
    # default: 600s
    # How long <command> <worker name> 1 and <command> <worker name> 0 may run before they are stopped. A command still running holds up new work for every worker.
    # any of
    [ <string>
    , <integer> # 64 bit signed integer
    ]
  tick: # optional
    enable: # optional
      # default: 'True'
      # Whether to run the tick command. Turn it off while autoscale.command cannot handle being called with only a worker name.
      <boolean>
    command: # optional
      # Command to run for every worker on every interval, as <command> <worker name>, with the worker's queue and activity in its environment. Defaults to autoscale.command.
      <string>
    interval: # optional
      # default: 60s
      # How long to wait between runs of the tick command. Every run queries the queue once per worker, so a short interval costs database time, and a long one leaves a pool short of machines for longer. At least 10s.
      # any of
      [ <string>
      , <integer> # 64 bit signed integer
      ]
    timeout: # optional
      # How long a tick command may run before it is stopped. A command still running holds up every later tick. Defaults to ten intervals.
      # any of
      [ <string>
      , <integer> # 64 bit signed integer
      ]