Skip to content

mac maintenance

Jump to bottom
Heitor Neiva edited this page Nov 3, 2025 · 13 revisions

Mac Maintenance

This document describes the manual steps required to maintain the Mac signers.
Until we fully automate setup and maintenance, maintenance must be performed by SSHing into each Mac.

The list of available machines can be found here. Please keep the machine statuses up to date on that page.

Before performing maintenance, consider quarantining the machine to prevent jobs from being scheduled on it.

Scriptworkers are currently installed to /usr/local/builds. See modules/roles_profiles/manifests/profiles/mac_v3_signing.pp for more information.

SSH Access

We use Duo for authentication on the Mac signers. To access a Mac via SSH, your account must be added to the Duo allowlist.

If you’d like to cache your ssh authentication for 10 minutes per host, add the following to your ~/.ssh/config:

Host dep-mac-v4-signing* adhoc-mac-v4-signing* fx-mac-v4-signing* tb-mac-v4-signing* vpn-mac-v4-signing*
    ControlMaster auto
    ControlPath ~/.ssh/ssh-%C
    ControlPersist 10m

Updating Python Packages

To update Python packages on the Mac signers:

  1. In the ronin_puppet repository, run:

    data/update-scriptworker-revisions.py

    This updates the workers to point to the latest scriptworker-scripts master revision.

  2. Create a pull request and merge it into the master branch.

  3. After merging, push master to the macos-signer-latest branch:

    git fetch && git push origin origin/master:macos-signer-latest

  4. The Mac signers automatically self-update every 15 minutes based on the macos-signer-latest branch.

Updating the Widevine Repository

The Widevine repository is not revision-pinned and is not updated automatically. To update it manually:

  1. Update the Widevine repository.

  2. Remove the existing Widevine directory on the signing workers:

    # ff-prod or tb-prod
rm -rf /usr/local/builds/scriptworker/widevine

# dep environments
rm -rf /usr/local/builds/{dep1,dep2,tb-dep,etc}/widevine

  3. When you run Puppet (see below), it will repopulate and reinstall Widevine with the latest code.

Pausing Periodic Puppet

If you’re performing maintenance on a Mac (remember to quarantine it first!), you may want to temporarily disable periodic Puppet runs.

You can do this in one of two ways:

  • Modify /usr/local/bin/periodic-puppet.sh to exit early before running Puppet.
  • Unload the com.mozilla.periodic.plist LaunchDaemon (see below).

Ensure Puppet is not already running before unloading it, otherwise it may revert your changes.

Unloading Agents and Puppet

There are two types of LaunchDaemons on the signers:

  1. Puppet periodic daemon: /Library/LaunchDaemons/com.mozilla.periodic.plist
  2. Scriptworker daemons: /Library/LaunchDaemons/org.mozilla.scriptworker.*.plist

To unload the Puppet periodic daemon:

launchctl unload /Library/LaunchDaemons/com.mozilla.periodic.plist

To unload all Scriptworker daemons:

for d in /Library/LaunchDaemons/org.mozilla.scriptworker.*.plist; do launchctl unload "$d"; done

Wiping Secrets

Secrets must be wiped whenever a machine is being moved, repurposed, or retired — or if there’s any chance it may have been compromised.

Secrets are stored in the certs/ directory inside the scriptworker install location.

When wiping secrets, remove the following files (using rm -P to securely overwrite them):

  • ed25519_privkey (production only)
  • *.keychain
  • widevine*.crt

Clone this wiki locally