Skip to content

Add readonly mode, more efficient package sets#760

Open
thomashoneyman wants to merge 1 commit intomasterfrom
trh/package-set-efficiency
Open

Add readonly mode, more efficient package sets#760
thomashoneyman wants to merge 1 commit intomasterfrom
trh/package-set-efficiency

Conversation

@thomashoneyman
Copy link
Member

This is a quick prototype pull request demonstrating two things:

  1. Introducing a READONLY env var that can be used if you want to test the live server but set it to not perform any writes (to storage, pursuit, the registry, etc.)
  2. Updating package sets to not always blow away the output directory, thus making it so incremental sets can be fast to compile instead of recompiling the full 500+ package sets.

We've been struggling with seeing package set builds time out in the registry server because it's a bit underpowered and we only allot 30 minutes total for a job. It's not able to get through the full set before timing out. We can bump the timeout, but then the package set job will block anyone from publishing for the full duration of the build.

So: why is it so slow? The UpgradeAtomic handler clears packages/, output/, and output-backup/ at the start of every job to ensure there are no false positive compilations. A full recompilation on a beefy machine may only take a minute or two, but on the registry server it's taking forever.

The WIP code does a more targeted wipe instead:

  1. output-backup/: Always wiped (this is transient rollback state from attemptPackage).
  2. output/: Only wiped when the compiler version changes, but otherwise preserved for incremental compilation. We're relying on the compiler here to detect changes to packages.
  3. packages/: Synchronized, in that extraneous directories are removed and missing ones are installed, so the packages directory matches the contents of the submitted package set job.

This should work; the compiler's incremental compilation takes these steps:

  1. Receives source globs: purs compile packages/**/*.purs
  2. Discovers all .purs files matching the globs
  3. For each source file, checks output/<Module>/externs.cbor timestamp vs source timestamp
  4. Skips recompilation if output is newer than source
  5. If a module's source is not in the glob, the module is considered to not exist regardless of the contents of output/

Separately, the syncPackages function figures out which directories in packages/ don't belong to the target set and removes them. Then installPackages fills in missing ones. This is equivalent to a wipe + reinstall but avoids re-downloading and re-extracting hundreds of tarballs.

Of course, it's hard to think through all possible edge cases that could be hit here with preserved files so I haven't marked this as a ready, but I wanted to get it here for discussion.

@thomashoneyman thomashoneyman marked this pull request as ready for review March 16, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant