Configuring compute

lc run and lc materialize only execute tasks through allocated compute resources, which may be on the local system or on a Slurm cluster. Available compute providers that your agents may request compute allocations from are configured in the ~/.lightcone/compute.yaml file which is described in this guide.

How the compute.yaml file worksCopy Anchor Link

By default, the compute providers are configured in ~/.lightcone/compute.yaml, but that location may be overidden by setting the environment variable LC_COMPUTE_CONFIG=/path/to/compute.yaml.

This file lists offers: the allocations you can request. Each offer names its provider, which says where the compute comes from: local (this machine) or slurm (the Slurm cluster your sbatch submits to). It also says how many CPUs, how much memory and how many GPUs each node has, how many nodes at most, and how long the allocation may run.

For example, this file lets you allocate up to four 64-CPU nodes on a Slurm cluster:

offers:
  - name: cpu                       # a name you choose
    provider: slurm                 # allocate compute by submitting Slurm jobs
    resources: {cpus: 64, memory: 240}   # each node: 64 CPUs and 240 GiB
    max_nodes: 4                    # at most 4 nodes per allocation
    time: {default: 1h, max: 12h}   # 1 hour unless you ask for more, 12 at most
    config:
      account: myproject            # the Slurm account to charge

With it, lc compute launch --cpus 64 --memory 240 --num-nodes 2 submits a one-hour Slurm job for two such nodes, charged to myproject. lc keeps its built-in local offer after yours, so lc compute launch without options still allocates this machine. Every field is explained in A first Slurm configuration.

When you run lc compute launch, lc goes through the offers in the order they appear in the file and launches the first one that matches what you asked for. You may inspect the currently configured resources by running:

lc compute resources          # the offers lc sees, in the order it tries them

On your own machineCopy Anchor Link

Without a configuration fileCopy Anchor Link

With no file, lc has one built-in offer named local: all of this machine's CPUs and memory, as a single node.

Launch it without options:

lc compute launch --wait              # prints the cluster's name: local
lc materialize local
lc compute down local                 # optional: it also stops by itself

A local cluster stops on its own after 30 minutes without work. You can only run one local cluster at a time on each machine.

Use part of the machine, or change when it stopsCopy Anchor Link

To change how much of the machine a local cluster uses, or when it stops, write your own local offer. It replaces the built-in one:

offers:
  - name: workstation
    provider: local
    resources: {cpus: 4, memory: 8}   # 4 CPUs and 8 GiB of memory
    max_nodes: 1                      # a local offer is always one node
    time: {idle: 2h, max: 8h}         # stop after 2 hours without work; --time up to 8 hours

lc compute launch without options now uses this offer, and still names the cluster local. A local cluster you started before writing the file stays visible, and can still be stopped.

resources leaves the rest of the machine free for other work. lc schedules recipes within these limits, but doesn't reserve them: other programs on the machine can still use the rest.

time sets when the cluster stops:

• idle stops the cluster after that long without work.
• default is a hard limit applied to every launch, which stops the cluster even mid-recipe.
• max is the longest hard limit you can ask for with lc compute launch --time.

A cluster stops at whichever limit it reaches first, and time needs an idle or a default. Durations combine days, hours, minutes and seconds, in that order: 45s, 30m, 1h30m, 2d.

Turn local compute offCopy Anchor Link

allow_local: false

Use this where work shouldn't run on the machine itself, such as a cluster's login node (see Keep work off the login node).

GPUsCopy Anchor Link

On Linux, local clusters can use this machine's GPUs. lc doesn't go looking for them on its own: it uses the GPUs listed in the CUDA_VISIBLE_DEVICES environment variable when you run lc compute launch. That keeps you in control of which GPUs your analyses use. Set the variable to the GPUs you'd like to share, in your shell or your shell profile; on some systems it's already set for you, for instance inside a Slurm GPU job. When it's unset or empty, local clusters run without GPUs.

There's nothing to add to compute.yaml. With two GPUs listed, lc compute resources shows the built-in offer with GPU:2, and a local cluster uses both of them. Launch it with --gpus 0 when you'd rather leave them out.

List exactly the GPUs you want the cluster to use. lc counts the devices in the variable, written as indexes such as 0,1 or as GPU-… UUIDs, and passes the variable on as it is. --gpus GPU:2 must match that count, so to use fewer GPUs, list fewer.

To name the GPU model, or to fix the CPUs and memory, list a local offer yourself:

offers:
  - name: workstation
    provider: local
    resources: {cpus: 16, memory: 64, accelerators: A100:2}
    max_nodes: 1
    time: {idle: 30m}

• It replaces the built-in offer, like any local offer, so it only has the GPUs it declares.
• A100:2 is a label you choose: lc doesn't detect your hardware. Launch the offer as it is with no --gpus, or ask for --gpus A100:2 or --gpus GPU:2 (any model). CUDA_VISIBLE_DEVICES must list at least two GPUs; --gpus 0 launches the offer without them.
• The offer has to fit on the machine. lc skips an offer with more CPUs or memory than the machine has. Adjust the numbers to your machine.

On a Slurm clusterCopy Anchor Link

On a Slurm cluster, lc compute launch submits a job, and your recipes run on the compute nodes it gets. The rest of the workflow doesn't change. You run lc from a login node, or anywhere else you can submit jobs from.

A first Slurm configurationCopy Anchor Link

offers:
  - name: cpu
    provider: slurm
    resources: {cpus: 64, memory: 240}
    max_nodes: 4
    time: {default: 1h, max: 12h}
    config:
      account: myproject
      partition: standard

• provider: slurm allocates by submitting Slurm jobs, to the cluster your Slurm commands reach by default: the one sbatch submits to from this shell.
• resources is what each node gets, here 64 CPUs and 240 GiB of memory; see Resources.
• max_nodes is the most nodes one launch can ask for.
• time.default is the job's time limit when you don't pass --time, and time.max the longest you can ask for. A Slurm offer needs a default and can't use idle: the cluster runs until its time limit or until lc compute down, so stop it when you're done.
• config holds the Slurm options for the job:

config also takes a few optional settings for the workers that run your recipes; see Advanced settings.

Put these settings in the offer, even if your shell already sets them: lc ignores Slurm settings from environment variables such as SBATCH_ACCOUNT, so a job doesn't depend on the shell it was launched from.

To launch, give the cluster a name, and the CPUs and memory per node, which select the offer:

lc compute launch --name analysis --cpus 64 --memory 240 --time 2h --wait
lc materialize analysis
lc compute down analysis

• Naming. --name analysis names the cluster, and the other commands refer to it by that name. Without --name, lc makes one up, such as lc-a1b2c3d4e5f6, and prints it. A name can be reused once its cluster has ended.
• More nodes. --num-nodes 2 asks for two nodes, up to the offer's max_nodes.
• Waiting. --wait waits for the job to start, but gives up after 5 minutes (the job stays queued). Pass --timeout with a number of seconds to wait longer, or wait again later with lc compute status analysis --wait.

An salloc job lives in a process on the login node, so it may end when you log out. Use the default sbatch unless your site requires salloc for the QOS you want.

While a Slurm allocation runs, keep at least one Slurm offer in the file. lc only looks for allocations through the providers your offers name: without one, lc compute status stops listing the allocation and its name stops working. Its full ID, the id in lc compute launch --json, still reaches it.

ResourcesCopy Anchor Link

An offer's resources describe one node of the allocation:

Slurm offers need memory in a whole number of MiB. lc compute launch picks an offer by these values:

• --cpus and --memory must match an offer exactly. Add + to accept any offer at least that large: --cpus 32+ --memory 100+ takes the first such offer in the file.
• --gpus must match the offer's GPU count exactly, and doesn't take +. Without it, only CPU-only offers match.

GPUsCopy Anchor Link

To give each node GPUs, add accelerators to an offer:

  - name: gpu
    provider: slurm
    resources: {cpus: 32, memory: 120, accelerators: GPU:4}
    max_nodes: 2
    time: {default: 1h, max: 6h}
    config:
      account: myproject
      partition: gpu

and ask for them with --gpus:

lc compute launch --name gpu-analysis --cpus 32 --memory 120 --gpus GPU:4 --wait

lc asks Slurm for four GPUs on each node, and Slurm decides which ones the job sees.

GPU:4 accepts any GPU model. If your site has several models and you want a specific one, name it and add its Slurm type as gpu_type. sinfo -o "%P %G" lists each partition's GPUs; in gpu:a100:4, the type is a100:

    resources: {cpus: 32, memory: 120, accelerators: A100:4}
    config:
      account: myproject
      gpu_type: a100

A named model always needs gpu_type, even when the two are spelled the same. Launch it with --gpus A100:4, or with --gpus GPU:4 to take whichever GPU offer comes first.

Fast and batch offersCopy Anchor Link

A Slurm job can wait in the queue for seconds or for hours, depending on its QOS or partition. To choose between a queue that starts quickly and one that may wait, list one offer for each and label them with startup:

offers:
  - name: cpu
    provider: slurm
    resources: {cpus: 64, memory: 240}
    max_nodes: 4
    time: {default: 1h, max: 12h}
    startup: batch                  # a regular queue: it may wait a while
    config: {account: myproject, qos: regular}
  - name: cpu-debug
    provider: slurm
    resources: {cpus: 64, memory: 240}
    max_nodes: 1
    time: {default: 30m, max: 30m}
    startup: fast                   # a debug queue: it usually starts within minutes
    config: {account: myproject, qos: debug}

Offers are tried in order, so lc compute launch --name analysis --cpus 64 --memory 240 takes the regular queue. Add --startup fast to take the debug queue instead.

Keep work off the login nodeCopy Anchor Link

If local compute is still enabled, lc compute launch without --cpus and --memory starts a local cluster on the login node. NERSC login nodes are recognized and refuse it automatically. Elsewhere, add allow_local: false to the file.

Your home directory is shared, so the same file also applies on compute nodes: with local compute off, you can't start a local cluster inside an interactive job either. If you want to, leave it on, and always pass --cpus and --memory on the login node.

Advanced settingsCopy Anchor Link

An offer's config also takes optional settings for the workers that run your recipes. The defaults suit a home directory shared by all nodes:

  - name: cpu
    provider: slurm
    resources: {cpus: 64, memory: 240}
    max_nodes: 4
    time: {default: 1h, max: 12h}
    config:
      account: myproject
      task_slots_per_node: 16
      scratch_root: /local/scratch

lc keeps each allocation's private files, such as the scheduler's address, its credentials and Slurm's logs, under ~/.lightcone/compute. The driver and every node must reach them. If your home directory isn't shared by every node, move them with the top-level connection_root:

connection_root: /shared/scratch/me/lightcone-compute

Stop running allocations before changing it: lc no longer finds those it started under the old directory.

ReferenceCopy Anchor Link

Every field of compute.yaml:

Memory can be written 240, 240GB or 240GiB, all meaning 240 GiB; Slurm offers need a whole number of MiB. Durations use d, h, m and s, in that order: 1h30m.

Examples on known Slurm computersCopy Anchor Link

These configurations show the shape of a setup on Slurm systems we know. Replace the account, and adjust the sizes to your allocation.

NERSC PerlmutterCopy Anchor Link

A configuration with a regular and an interactive CPU queue, and a GPU queue. It hasn't been tested by submitting jobs at NERSC.

offers:
  - name: cpu
    provider: slurm
    resources: {cpus: 256, memory: 480}
    max_nodes: 16
    time: {default: 1h, max: 12h}
    startup: batch
    config: {account: myproject, constraint: cpu, qos: regular}
  - name: cpu-interactive
    provider: slurm
    resources: {cpus: 256, memory: 480}
    max_nodes: 2
    time: {default: 1h, max: 4h}
    startup: fast
    config: {submit: salloc, account: myproject, constraint: cpu, qos: interactive}
  - name: gpu
    provider: slurm
    resources: {cpus: 32, memory: 128, accelerators: GPU:4}
    max_nodes: 2
    time: {default: 1h, max: 4h}
    startup: batch
    config: {account: myproject, constraint: gpu, qos: regular}

The two CPU offers have the same shape, so lc compute launch --cpus 256 --memory 480 takes the first one, in the regular queue. Add --startup fast to take the interactive one instead.

A few things specific to NERSC:

• Move uv's cache to scratch before launching. Recipes run through uv, which locks files in its cache, and Perlmutter's compute nodes can't lock files in $HOME, where the cache lives by default. Add this to your shell profile; a cluster launched without it has to be relaunched:

  Copy Text

  export UV_CACHE_DIR=$PSCRATCH/uv-cache

• Leave partition out. NERSC picks the partition from the QOS and constraint.

• Login nodes refuse local compute. An interactive job on a compute node can still use the built-in local offer.

• Containerized projects use podman-hpc, which NERSC provides. It's also the only container runtime that gives recipes access to GPUs.