Core concepts

This page explains how lc works: what a project contains, what it records about each output, and how it decides what to remake. You don't need it to use the CLI.

What a project containsCopy Anchor Link

A project is a DataLad dataset, a git repository with git-annex, that holds an ASTRA specification and a uv project:

• astra.yaml describes the analysis: its inputs, outputs, recipes and decisions. lc reads everything it runs from this file.
• pyproject.toml and uv.lock describe the environment: the packages recipes can import, at exact versions. lc builds .venv from the lock. The lock is committed, and .venv isn't.
• The repository holds the code, the data and the results, with their history. git stores the code and the specification, and git-annex the data and results, as the next section explains.

Everything about the analysis is in the project. The only lc configuration outside it, ~/.lightcone/compute.yaml, describes the machines you run on. A clone has everything except .venv and the local git-annex setup, which lc init rebuilds.

Add dependencies with uv:

uv add numpy

This updates pyproject.toml, re-locks and syncs .venv. Recipes can import only from the locked environment, so packages installed elsewhere on your machine, with pip install for example, don't affect them.

A project is a DataLad datasetCopy Anchor Link

A git repository with git-annex has the structure of a DataLad dataset, and lc init writes the one thing that makes it a dataset: an ID in .datalad/config. lc itself runs only git and git-annex, so you don't need DataLad, but DataLad commands work on the project: datalad clone, datalad get, datalad rerun.

What git stores, and what git-annex storesCopy Anchor Link

.gitattributes, written by lc init, sends data and results to git-annex and keeps everything else in git:

Annexed files stay ordinary files in your working tree, not symlinks: git-annex's unlocked mode, through the filter=annex attribute. A plain git add hands their content to git-annex, which keeps it in .git/annex/ and commits a small pointer in its place. lc init also sets filter.annex.required, so that a git add run without git-annex on the PATH fails, rather than silently committing the data into git itself.

Inside an existing git repository, lc init doesn't create a nested one: the project becomes a directory of the enclosing repository, and git-annex is set up there.

Run recordsCopy Anchor Link

Each output's commit message is a run record in DataLad's format: the command that made the output, its inputs, and its outputs.

[DATALAD RUNCMD] fit [baseline]

=== Do not change lines below ===
{
 "chain": [],
 "cmd": "uv run --no-project --with 'lightcone-cli==0.5.0' -- python -m lightcone.engine.worker baseline/fit",
 "dsid": "2c1d341c-0dae-4827-a97a-5510eb672ad3",
 "exit": 0,
 "inputs": [
  "data/points.csv"
 ],
 "outputs": [
  "results/baseline/fit.json",
  "results/baseline/.fit.manifest.json"
 ],
 "pwd": "."
}
^^^ Do not change lines above ^^^

The command runs the same version of lc that made the output, which rebuilds the environment from that commit's uv.lock. So on any clone, datalad rerun 7fa75c1 fetches the inputs, remakes the output and commits it again. A containerized run also lists its image, which datalad rerun fetches too. lc build registers each image in .datalad/config the way datalad-container does.

Clones and sharingCopy Anchor Link

A clone carries the history, the spec, the code and every manifest, but only pointers for annexed files. git annex get (or datalad get) fetches a file's content when you need it, and lc materialize fetches the declared inputs it needs on its own. lc status checks each output's inputs against its manifest, so it needs the inputs' content, and says so when they are missing; it never needs the results themselves. Fetch the inputs, and lc status tells you whether every result is current without downloading any of them.

git-annex fetches content from wherever a copy is recorded. To share a project, push the git history to any git host, and the annexed content to a place that can store it: GitHub and similar hosts take only the git history, so use a server reached over SSH, a host that supports git-annex, or one of git-annex's special remotes, such as an S3 bucket. datalad push and git annex copy --to send the content there.

What each output recordsCopy Anchor Link

Next to each output, lc writes a manifest, .<output_id>.manifest.json, with three hashes:

1. The definition: the output's recipe and the decision values it uses.
2. The inputs: the content of each declared input.
3. The environment: the lock, the Python interpreter, and the container image if the project declares one.

The manifest also records the git commit the run started from. lc status and lc materialize compare these hashes with the project to give each output a state:

A stale output no longer matches the analysis, so lc materialize remakes it. A behind output still does; only the environment changed. That happens often: adding a package for a plotting script re-locks the whole project, and remaking days of computation for that would be wasted. The manifest records the commit the output was made at, and that commit's uv.lock rebuilds its environment if you need it.

To remake behind outputs as well, before a release for example, add --refresh:

lc materialize "$CLUSTER" --refresh

--refresh doesn't remake current outputs, and no option skips stale ones.

An output edited by hand is stale too. Each output is committed by the run that made it, so a later commit that changes it isn't a run record: lc status reports the output as stale and names that commit.

How results are committedCopy Anchor Link

lc commits results to the project's repository, where git-annex stores them. Two rules follow:

• Commit before you run. lc materialize refuses to start with uncommitted changes, because each output records the commit of the code that made it.
• Each output is committed as it's made, with its manifest, in a commit whose message is a run record. If a recipe fails, its partial output is removed. git log shows every run.

Only lc writes to results/. Don't add or edit files there yourself: they would have no manifest or run record.

Direct and containerized executionCopy Anchor Link

The project decides how recipes run; there is no setting for it.

• Direct (the default): recipes run on the host, in the project's .venv, inside a sandbox: Landlock on Linux, Seatbelt on macOS. The project is read-only to them, except the directory their output goes to, and tools the project doesn't declare don't run.
• Containerized: declaring [tool.lightcone.image] in pyproject.toml makes recipes run in a container image built from that declaration. lc build commits the image to the repository, so a clone gets the exact image without a registry or credentials. lc status shows the mode and whether the image is built.

Each manifest records, in its hermeticity field, the isolation the run actually had. On a host with no sandbox, the recipe still runs, and its manifest says so.

Checking a projectCopy Anchor Link

• lc status shows each output's state and the commit it was made at. It runs nothing, works with uncommitted changes, and always exits 0.
• lc materialize --check runs nothing either, and exits 1 if a run would make anything. Use it in scripts and CI. It covers only outputs that have a recipe: lc status lists any declared without one as no recipe.

Both take --json. In the check's report, up_to_date is the key to test.

PublishingCopy Anchor Link

Declaring a license under [project] in pyproject.toml marks the project for publication. From then on, each lc materialize keeps an RO-Crate, ro-crate-metadata.json, at the project root: a description of the project, its outputs and the runs that made them. The repository is the crate, so to deposit the project, archive the repository, with git archive for example.

Where to nextCopy Anchor Link

• Configuring compute: running on Slurm, and with GPUs.
• Troubleshooting: common problems, and how to solve them.