NERSC
Set up the Lightcone Stack on Perlmutter and run analyses on its compute nodes.
Run these steps once, on a Perlmutter login node.
Install uv
NERSC doesn't provide uv, but it installs into your home directory:
curl -LsSf https://astral.sh/uv/install.sh | shuv lands in ~/.local/bin; make sure that directory is on your PATH. The stack needs uv 0.12 or
later: check with uv --version. git is already installed.
Move uv's cache to scratch
Recipes run through uv, which locks files in its cache, and Perlmutter's compute nodes can't lock
files in $HOME, where the cache lives by default. Add this line to your ~/.bashrc, then open a new
shell:
export UV_CACHE_DIR=$PSCRATCH/uv-cacheDo this before you launch any compute: an allocation started without it has to be relaunched.
Install the Lightcone CLI
With uv's cache on scratch, install the CLI, which provides the lc command:
uv tool install lightcone-cliThe install also puts git-annex on your PATH; if lc runs but git-annex doesn't, run uv tool update-shell and open a
new shell. See Install the CLI for more.
Install your agent and the plugin
Install Claude Code or Codex as NERSC's own guides describe, then add the Lightcone plugin as in the Quick Start. The plugin checks that the CLI is the version it needs when you start a project.
Describe Perlmutter's queues
The CLI launches compute from the offers in ~/.lightcone/compute.yaml. This one has a regular and
an interactive CPU queue, and a GPU queue:
offers:
- name: cpu
provider: slurm
resources: {cpus: 256, memory: 480}
max_nodes: 16
time: {default: 1h, max: 12h}
startup: batch
config: {account: myproject, constraint: cpu, qos: regular}
- name: cpu-interactive
provider: slurm
resources: {cpus: 256, memory: 480}
max_nodes: 2
time: {default: 1h, max: 4h}
startup: fast
config: {submit: salloc, account: myproject, constraint: cpu, qos: interactive}
- name: gpu
provider: slurm
resources: {cpus: 32, memory: 128, accelerators: GPU:4}
max_nodes: 2
time: {default: 1h, max: 4h}
startup: batch
config: {account: myproject, constraint: gpu, qos: regular}Replace myproject with your NERSC project, such as m1234. The two CPU offers have the same shape, so
lc compute launch --cpus 256 --memory 480 takes the regular queue, and --startup fast the
interactive one. Configuring compute explains every field.
Then check what the CLI sees:
lc compute resourcesNot yet tested at NERSC
This configuration hasn't been tested by submitting jobs at NERSC yet. If a launch fails, tell us on Discord.
Where to work
From a login node. Start your agent in your project on a login node, where NERSC recommends running coding agents. Login nodes limit each user to 30 GB of memory and 12.5% of the CPUs, and refuse local compute, so your recipes run in Slurm allocations the CLI launches from your offers. Each allocation queues like any job: the interactive offer usually starts within minutes, the regular one may wait.
Inside an allocation. To choose the node yourself and skip the queue between steps, open an interactive allocation and start your agent on the compute node:
salloc --nodes 1 --qos interactive --time 01:00:00 --constraint cpu --account m1234The CLI then runs recipes on that node through its built-in local offer, which uses one node, so ask
for one. For GPUs, use --constraint gpu --gpus 4; the CLI uses the GPUs listed
in CUDA_VISIBLE_DEVICES. An interactive allocation
lasts at most 4 hours, and NERSC cancels the request if no node is free within 6 minutes. Your agent
must also be able to reach its service from the compute node.
Codex file locking on compute nodes
Codex uses file locks under ~/.codex by default, but Perlmutter's compute nodes
do not support file locking on $HOME.
This can cause errors when running Codex inside an interactive allocation. To keep Codex's state
off $HOME, run this before starting Codex:
export CODEX_HOME="${PSCRATCH:?PSCRATCH is not set}/codex"
mkdir -p "$CODEX_HOME"
chmod 700 "$CODEX_HOME"Reuse this location across jobs to retain session history; add the export to ~/.bashrc to make
it persistent. Existing settings and history are not migrated automatically, and you may need to
sign in again. Back up important sessions: scratch files may be purged after 8 weeks without access.